Supported Models

Whether MaruKVConnector can store and reuse a model’s KV cache in vLLM depends on the structure of the KV cache that vLLM builds for the model. This page lists the structures it stores and the ones it refuses; a structure or setting that is not listed has not been tested. Support for a range of models on SGLang HiCache and NVIDIA Dynamo, as on vLLM, is planned.

On the CXL storage backend (the default):

KV cache structure	Models	Supported status		Tested with
		Single GPU	Tensor parallel
Full attention (MHA, GQA), one KV cache group	Qwen3 (dense and MoE), GLM-4.5, GLM-4.6	🟢	🟢	Llama-3.1-8B
Sliding window + full attention	gpt-oss	🟢	🟢	gpt-oss-20b
Gated DeltaNet (Mamba-type) + full attention	Qwen3.5, Qwen3.6, Qwen3.8, Qwen3-Next	🟢	🟢	Qwen3.5-0.8B
Attention groups whose KV heads differ in count or size	Gemma 4, MiMo-V2	❌	❌
DSA: MLA latent + indexer cache	DeepSeek-V3.2, GLM-5	❌	❌
MLA with other kinds of group	GLM-5.3-Flash, DeepSeek-V4.1-Flash	❌	❌
Chunked local + full attention	Llama 4	❌	❌

🟢 means that Maru stores and loads the KV cache of such models, and ❌ that the connector refuses them at startup. Models other than the one in “Tested with” share their row’s KV cache structure but have not been run. Models with several KV cache groups also need the settings in Hybrid models.

Pipeline parallelism and decode or prefill context parallelism are refused at startup. See Tensor parallelism for how each rank stores its part of the KV cache.

Storage backends

The table holds for the cxl backend. The cpu, mixed and remote_cxl backends take only text-only models with one KV cache group and without MLA, at TP=1 and without data parallelism, and refuse LoRA and quantized KV caches.

Reporting results

If you run a model that this page does not list, please open an issue with the model, the versions of vLLM and Maru, the parallel settings and the result.