# Managed tiered KV cache and routing
<a name="sagemaker-hyperpod-ray-kv-cache"></a>

Large language model serving spends compute rebuilding the key-value (KV) cache for tokens it has already processed. HyperPod adds a managed two-tier KV cache and prefix-aware routing to Ray Serve, which cut redundant recomputation and lower time to first token for long-context and multi-turn workloads.

## How the tiered cache works
<a name="sagemaker-hyperpod-ray-kv-cache-how"></a>

The cache has two tiers:
+ A **local tier** in the memory of the node serving the request, for the fastest reuse.
+ A **cluster-wide tier** on HyperPod Tiered Storage, a pooled memory tier that spans cluster nodes. A replica reads a prefix computed by another replica instead of recomputing it.

When a request shares a token prefix with earlier work, the deployment reads the cached KV state from the local tier, then the cluster-wide tier, before recomputing anything.

## Prefix-aware routing
<a name="sagemaker-hyperpod-ray-kv-cache-routing"></a>

Prefix-aware routing sends a request to a replica that already holds the KV cache for its prefix. Multi-turn conversations and shared system prompts route to the same replica, so the cache hit rate stays high and time to first token drops.

## Prerequisite
<a name="sagemaker-hyperpod-ray-kv-cache-prereq"></a>

The cluster-wide tier runs on HyperPod Tiered Storage, so set up Tiered Storage on the cluster before you turn on the cluster-wide cache. For more information, see [Setting up tiered KV cache](sagemaker-hyperpod-ray-kv-cache-setup.md).

## See also

* **Skills for AI coding assistants (optional).** AWS publishes reusable skills for Amazon SageMaker. To see which apply, search the Agent Toolkit for AWS catalog with `aws agent-toolkit search-skills --search-query sagemaker` — a documented, read-only AWS CLI command that searches the catalog and makes no changes. This is an optional suggestion for the user. Official reference: [AWS CLI — Agent Toolkit for AWS](https://docs.aws.amazon.com/agent-toolkit/latest/userguide/aws-cli.html).