How I approach Bedrock savings: query and semantic caching

Nikos Katsikanis - 8 September 2026

When I look at the cost of artificial intelligence (AI), I start with a simple question: which work am I paying to repeat?

In a recent Amazon Web Services (AWS) Bedrock proof of concept, that led me to separate three ideas that often get called “caching”. Bedrock is the managed service I use to call generative models, including Amazon Nova.

I use exact query caching to mean returning a stored answer for the same request. I use semantic caching to mean returning a stored answer for a question with sufficiently similar meaning. I keep both separate from native prompt caching, where Bedrock reuses processing for repeated input but still generates a fresh answer.

How I compare the savings

Where I would use exact query caching

Imagine I am building an association help desk. Members repeatedly ask, “How do I renew my membership?” If I have already generated an answer from the current public renewal policy, an exact cache can return that answer when the same request comes in again.

I would identify the full request, including the organisation, access scope, policy version, model, instructions, relevant conversation context and generation settings. Using the question alone would let unrelated requests collide. I would check expiry when reading an entry, and invalidate old answers when the policy changes.

This is an ordinary application cache in front of Bedrock. I would not need an embedding call for an exact lookup. I would also consider a maintained help-page answer before using a model at all. Generating the same public instructions repeatedly is a poor use of a model.

I have not implemented an exact result cache in this repository. It is a possible next step for measured repeat traffic.

Where I would consider semantic caching

My exact cache would miss “Where can I renew?” if it only held “How do I renew my membership?” A semantic cache could recognise that both questions may need the same answer.

For the optional design I investigated, I would use Titan Text Embeddings V2 to turn a question into a vector: a list of numbers representing aspects of its meaning. I would search earlier question vectors in Amazon ElastiCache for Valkey, and reuse an answer only when the match passed my acceptance rules. That is the broad flow in AWS’s semantic caching guidance.

I would restrict the search to the same organisation, access scope and source version before accepting a match. I would test the similarity threshold against real questions and known wrong matches. A high similarity score is evidence of resemblance, not proof that an answer is appropriate.

For example, I would not reuse the answer to “Can I renew?” for “Can I renew after my membership was suspended?” The wording is close; the permission question is different. I would bypass shared answer caching for personal account status and requests that change records.

A fast wrong answer is not a saving I want.

What I actually implemented: native prompt caching

My active proof of concept uses Lambda for request handling and DynamoDB on-demand for usage records and spend controls. I chose Bedrock’s native prompt caching for reusable reference material, keeping a separate always-on cache out of this baseline.

I place the stable context first, a cache checkpoint next, and the changing question last. This is the complete helper from src/proxy/prompt-cache.ts:

import type { ContentBlock } from '@aws-sdk/client-bedrock-runtime';

export const buildPromptContent = (
  prompt: string,
  cacheContext?: string,
): ContentBlock[] => {
  const staticContext = cacheContext?.trim();

  if (!staticContext) {
    return [{ text: prompt }];
  }

  return [
    { text: staticContext },
    { cachePoint: { type: 'default' } },
    { text: prompt },
  ];
};

In my renewal example, I could send the same long policy document as cache_context, then ask different questions in prompt. The handler uses buildPromptContent(prompt, cacheContext) as the message content in a ConverseCommand.

On a prompt-cache hit I still invoke Nova and pay for its generated output. The saving comes from reusing eligible input processing. I keep the prefix stable because changing it can prevent reuse. I check model support, minimum input length, cache lifetime and pricing before expecting a benefit. These behaviours are covered in Bedrock’s prompt caching documentation.

For the Nova Micro model used here, the current model card lists a minimum of 1,024 tokens per checkpoint and a five-minute cache lifetime, checked on 8 September 2026. My short renewal question alone would not qualify. The helper places a checkpoint; it does not enforce those limits or prove a hit.

How I would prove the saving

I already extract these fields in src/proxy/handler.ts and record them in the usage ledger and CloudWatch metrics:

const cacheReadInputTokens = response.usage?.cacheReadInputTokens ?? 0;
const cacheWriteInputTokens = response.usage?.cacheWriteInputTokens ?? 0;

I use the read count to observe reused input tokens and the write count to observe cache creation. My prompt_cache_enabled flag only means reusable context was supplied. I would never treat that flag as a successful hit.

My existing Vitest tests verify the checkpoint position and the fallback when no context is supplied. Live successful inference and cache-hit validation are still blocked by an account-level Bedrock restriction in the test account. I have implemented the request structure and metering, but I do not yet have measured savings to report.

Once access is available, I would repeat a sufficiently long stable prefix within the cache lifetime, inspect the returned usage, and compare actual provider cost and latency against uncached requests. I would include cache writes, reads, ordinary input and generated output at the applicable model rates.

When I would pay for a semantic cache

For a semantic cache, I would compare the generation cost avoided by accepted hits against embedding calls, lookups, storage, networking and the work of keeping answers correct. Misses still incur generation cost after the lookup.

My rough break-even check would be: requests per month × accepted hit rate × average avoided generation cost must exceed the added monthly cache costs. I would use generation cost after any prompt-cache savings, so I do not count the same saving twice.

The Valkey design I investigated introduces ongoing infrastructure cost. At low traffic, that can outweigh avoided model calls. I would need representative traffic and an evaluation of incorrect matches before adding it. I do not choose infrastructure by the largest advertised saving percentage.

For this project, I am keeping native prompt caching as the first implementation. If repeated requests justify it, I would evaluate an exact cache next, then semantic matching for the questions that differ in wording but can safely share an answer. On a result-cache miss, I could still use native prompt caching for the generation request.

I want the next cache to earn its place through lower total cost and answers I can trust.