vLLM

Prerequisites

vLLM v0.14+ — required for KVConnectorBase_V1 support:

uv pip install vllm

See vLLM installation docs for GPU-specific options.

Integration Architecture

MaruKVConnector is a native vLLM KV connector that enables direct KV cache sharing between vLLM instances through CXL shared memory — without any middleware.

        flowchart LR
    subgraph prev["Previous (via LMCache)"]
        direction TB
        V1["vLLM"] --> LC["LMCacheConnector"] --> LE["LMCache Engine"]
        LE --> SM["StorageManager"] --> MC["MaruConnector"]
        MC --> MH1["MaruHandler"] --> CXL1["CXL"]
    end

    subgraph direct["Direct (this connector)"]
        direction TB
        V2["vLLM"] --> MKV["MaruKVConnector"]
        MKV --> MH2["MaruHandler"] --> CXL2["CXL"]
    end

    prev --> direct
    

By removing the LMCache middleware layer, the direct connector achieves:

• Fewer dependencies — only vLLM + Maru

• Zero-copy save path — GPU → CXL via single cudaMemcpy (no intermediate CPU buffer)

• Zero-copy load path — CXL mmap (CUDA pinned) → GPU via DMA

• No serialization overhead — raw tensor bytes, no MemoryObj conversion

Component Roles

MaruKVConnector implements vLLM’s KVConnectorBase_V1 interface with a dual-role design:

Role	Component	Responsibility
SCHEDULER	MaruSchedulerConnector	Checks chunk-by-chunk which prefix is cached; builds metadata for worker
WORKER	MaruWorkerConnector	Performs actual GPU ↔ CXL data transfers per chunk per layer

Both roles share the same MaruHandler connection to CXL shared memory.

Data Path

Store Path (GPU → CXL)

When a vLLM instance completes prefill, the connector stores KV cache in chunks:

        sequenceDiagram
    participant vLLM as vLLM Worker
    participant MKV as MaruKVConnector
    participant MH as MaruHandler
    participant MS as MaruServer
    participant CXL as CXL Memory

    vLLM->>MKV: save_kv_layer(layer, kv_tensor, attn_metadata)
    loop For each chunk (256 tokens)
        MKV->>MH: alloc(nbytes)
        MH-->>MKV: handle (CXL page)
        MKV->>CXL: dst.copy_(gpu_tensor) — single cudaMemcpy
        MKV->>MH: store(key, handle=handle) — register only, no memcpy
        MH->>MS: register_kv(key, region_id, offset, length)
    end
    

The save path uses handler.alloc() to get a pre-mapped CXL buffer, then copies GPU tensor data directly into it via torch.Tensor.copy_(). The subsequent store(handle=) call only registers the key in the metadata server — no additional data copy occurs.

Load Path (CXL → GPU)

When a second instance receives a request with a matching prefix:

        sequenceDiagram
    participant Sched as Scheduler
    participant MKV as MaruKVConnector
    participant MH as MaruHandler
    participant MS as MaruServer
    participant CXL as CXL Memory
    participant GPU as GPU Memory

    Sched->>MKV: get_num_new_matched_tokens()
    MKV->>MH: exists(chunk_key) per chunk
    MH->>MS: lookup_kv(key)
    MH-->>MKV: hit count (e.g., 3 of 4 chunks)

    Note over MKV: Worker phase
    MKV->>MH: retrieve(key) per chunk per layer
    MH-->>MKV: MemoryInfo (CXL mmap memoryview)
    MKV->>MKV: torch.frombuffer(info.view, dtype)
    MKV->>GPU: .to(device) — CXL→GPU DMA (pinned via cudaHostRegister)
    MKV->>MKV: Inject into KV cache layer via slot mapping
    

The CXL mmap region is pinned via cudaHostRegister by MaruHandler’s DaxMapper, so .to(device) triggers a direct DMA transfer from CXL to GPU memory without any intermediate CPU copy.

Chunk-Based Storage

Tokens are divided into fixed-size chunks (default 256 tokens) for storage:

Prompt: [tok0..tok255 | tok256..tok511 | tok512..tok767 | tok768..tok900]
         chunk 0        chunk 1          chunk 2          (incomplete, not stored)

Each chunk key = kv_{hash(tok0..end)}_L{layer} — a rolling prefix hash that encodes the full context up to that chunk, enabling partial prefix reuse.

Partial Prefix Reuse

Instance A:
  Request: "The quick brown fox jumps over the lazy dog. Once upon a time..."
  → Stores chunk 0, 1, 2

Instance B:
  Request: "The quick brown fox jumps over the lazy dog. In a galaxy far away..."
  → chunk 0, 1 hit (common prefix), chunk 2 miss
  → Loads chunk 0, 1 from CXL, computes the rest

Setup

Start Maru server:

maru-server
# Listens on tcp://0.0.0.0:5555 by default

Launch vLLM with MaruKVConnector (dynamic loading):

vllm serve <model> \
    --kv-transfer-config '{
        "kv_connector": "MaruKVConnector",
        "kv_connector_module_path": "maru_vllm",
        "kv_role": "kv_both",
        "kv_connector_extra_config": {
            "maru_server_url": "tcp://localhost:5555",
            "maru_pool_size": "4G"
        }
    }'

The kv_connector_module_path tells vLLM to dynamically import MaruKVConnector from the maru_vllm package. No vLLM source code changes are required.

Second instance (same node):

vllm serve <model> \
    --port 8001 \
    --kv-transfer-config '{
        "kv_connector": "MaruKVConnector",
        "kv_connector_module_path": "maru_vllm",
        "kv_role": "kv_both",
        "kv_connector_extra_config": {
            "maru_server_url": "tcp://localhost:5555",
            "maru_pool_size": "4G"
        }
    }'

Configuration

Settings in kv_connector_extra_config:

Parameter	Type	Default	Description
maru_server_url	str	tcp://localhost:5555	MaruServer address
maru_pool_size	str/int	1G	CXL memory pool size (4G, 500M, etc.)
maru_chunk_size	str/int	4M	Maru page size (CXL allocation unit)
maru_instance_id	str	auto	Unique instance ID (default: auto-generated UUID)
maru_eager_map	bool	true	Pre-map other instances’ CXL regions on connect
maru_kv_chunk_tokens	int	256	KV cache chunk granularity (in tokens)

Asynchronous transfer settings, all opt-in:

Parameter	Type	Default	Description
maru_async_load	bool	false	Load cache hits on a background thread between steps instead of inside the forward pass
maru_async_store	bool	false	Complete the store after the forward pass instead of on the last attention layer
maru_overlap_load_with_compute	bool	false	Overlap a packed load’s per-layer transfers with attention compute
maru_overlap_release_after_layers	int	1	Layers that must be copied before an overlapped load is reported complete and vLLM may schedule the request; requires maru_overlap_load_with_compute

Storage format — how a request’s KV is grouped into CXL objects:

Parameter	Type	Default	Description
maru_use_layerwise	bool	false	false = chunkwise: one object per chunk holding every layer. true = layerwise: one object per (chunk, layer)

Diagnostics and fallback guards — leave these at their defaults in normal operation:

Parameter	Type	Default	Description
maru_load_admission_window	int	0	Cap on asynchronous loads in flight; 0 submits all
maru_log_timing	bool	false	Emit per-request timing diagnostics to stderr

Renamed parameters

These three knobs were renamed to name the axis a deployer chooses. The former names are still accepted and log a deprecation warning:

Former name	Current name
maru_enable_deferred_loading	maru_async_load
maru_enable_write_behind	maru_async_store
maru_enable_layerwise_overlap	maru_overlap_load_with_compute

maru_enable_async_loading and maru_enable_fused_load were removed. They gated a load path that the packed storage layout never entered, and no measurement ever exercised it.

Asynchronous load and store

By default both the cache-hit load and the populate store run to completion inside the model worker’s forward pass. The two settings below move that work off the critical path; they are independent and can be enabled separately.

maru_async_load parks a cache-hit request while a background thread performs the Maru lookup and the CXL→GPU transfer, then reports completion through a CUDA event. The forward pass no longer waits on the retrieve RPC. This is the mechanism vLLM itself calls an asynchronous load: the request waits in WAITING_FOR_REMOTE_KVS.

maru_async_store lets the store finish after the forward pass returns, so the first token of the current step is not delayed by the GPU→CXL copy and the metadata registration.

maru_load_admission_window bounds how many asynchronous loads may be in flight at once. The default 0 submits every load immediately. Set a positive value only if you need request-level backpressure.

Storage granularity and overlap

maru_use_layerwise selects how a request’s KV is grouped into CXL objects. The two layouts differ throughout the store and load paths:

	chunkwise (false, default)	layerwise (true)
One CXL object holds	every layer of one chunk	one (chunk, layer) pair
Key	<chunk_key>	<chunk_key>_L<layer_idx>
Keys per request	chunks	chunks x layers
CXL page size	per-layer size x layers	per-layer size
Completion marker	the chunk key itself, registered once every layer is written	a separate _DONE key
Store	one gathered D2H per chunk	one write per layer
Load	whole slab per chunk, contiguous pages coalesced	one retrieve per (layer, chunk)

The key count is the practical difference: a 64k prompt on a 32-layer model resolves 59 keys chunkwise versus 1,888 layerwise, and that ratio carries straight into retrieve metadata RPC volume. Chunkwise is the default for that reason; layerwise remains available for deployments that need per-layer object granularity.

maru_overlap_load_with_compute applies only to the packed layout and requires maru_async_load; it pipelines a request’s per-layer transfers against attention compute. The connector logs a warning and disables it when those prerequisites are not met.

The loader thread queues the per-layer copies while the request is still parked. The load is reported complete — which is what lets vLLM schedule the request — once maru_overlap_release_after_layers of them have landed, one layer by default, so the request’s prefill compute runs while the remaining layers arrive rather than after them. A cache-hit request computes only the tokens left over past its cached chunks, so one layer of compute is shorter than one layer of transfer: the compute fits inside the transfer and the transfer time is what sets time to first token. All parked-request transfers share one stream. That keeps each at full CXL bandwidth — splitting them across streams makes each transfer take as many times longer as there are loading requests, raising that floor — and it means a later request’s first layer only lands once the earlier one has finished, so requests take their turn without any extra admission mechanism.

The knob therefore applies at any concurrency. On a 16k prompt it lowered cache-hit TTFT by 24%, 25% and 17% at 2, 4 and 8 concurrent requests, and raised throughput throughout; per-token generation time rose 4-5% at 4 and 8 concurrent requests, which is the cost of every request starting earlier.

Choosing the release point

maru_overlap_release_after_layers sets how many layers must be copied before the load is reported complete. The layers still in flight at that point are waited for inside the request’s own forward pass, and those waits are taken on the model stream, which carries the whole batched step — so reporting early charges one request’s remaining load time to every request scheduled alongside it.

Waiting for more layers first keeps that cost out of the batch: while the request is unscheduled it is the only thing waiting on its own load. It does not delay the request’s own first token either, which needs the last layer no matter when the request was scheduled — the hold is free while it stays inside the delay the scheduler takes to re-admit the request anyway, and only past that does the count begin costing time to first token.

Which way to err depends on whether a layer’s copy or a layer’s compute takes longer:

• Compute slower than the copy — the forward pass never catches up, there is nothing left to wait on, and the default of 1 is already right.

• Copy slower than compute — the forward pass catches up almost at once and blocks on layers still in flight. Raising the count trades those blocks away.

Longer prompts and more concurrent loads both push a deployment toward the second case. The value is an absolute layer count, not a fraction of model depth, so one tuned on a 32-layer model does not carry to an 80-layer one, and a value at or above a model’s KV-layer count reports the load only after the last layer — the overlap turned off in all but name. Sweep it per deployment.

vllm serve <model> \
    --kv-transfer-config '{
        "kv_connector": "MaruKVConnector",
        "kv_connector_module_path": "maru_vllm",
        "kv_role": "kv_both",
        "kv_connector_extra_config": {
            "maru_server_url": "tcp://localhost:5555",
            "maru_pool_size": "4G",
            "maru_async_load": true,
            "maru_overlap_load_with_compute": true,
            "maru_overlap_release_after_layers": 7
        }
    }'

The count is read only by the overlap path, so maru_overlap_load_with_compute must be on for it to mean anything — along with the overlap’s own prerequisites, maru_async_load enabled and chunkwise storage (maru_use_layerwise left at false). Raising the count on its own changes nothing; the connector logs a warning saying so, and a separate warning if the overlap was requested but its prerequisites are unmet.

maru_kv_chunk_tokens

Controls how many tokens per chunk when storing KV cache:

• Smaller (64, 128): Finer prefix reuse granularity, more maru keys

• Larger (512, 1024): Fewer keys, but coarser reuse granularity

• Default 256: Good balance for most use cases

• Auto-aligned: Automatically adjusted to a multiple of vLLM block_size

maru_pool_size

CXL memory allocated per instance. Capacity estimation:

pool_size ≈ num_layers × kv_head_dim × num_kv_heads × 2(K+V) × max_cached_tokens × dtype_bytes

Example (Llama 7B, fp16):

32 layers × 128 head_dim × 32 heads × 2(K+V) × 4096 tokens × 2 bytes ≈ 2GB

Comparison with LMCache Path

Aspect	Via LMCache	Direct (this connector)
Dependencies	vLLM + LMCache + maru	vLLM + maru
Middleware	LMCache Engine, StorageManager, RemoteBackend	None
Serialization	LMCache MemoryObj conversion	torch tensor ↔ bytes direct
Prefix matching	LMCache CacheEngineKey hashing	vLLM token prefix hashing
Configuration	LMCACHE_CONFIG_FILE YAML	kv_connector_extra_config JSON
Save path	GPU → CPU → bytes → alloc + memcpy → CXL	GPU → CXL (single DMA via alloc)
Load path	CXL → clone → CPU → GPU	CXL → GPU (single DMA, pinned)

Troubleshooting

MaruServer Connection Failure

ERROR: Failed to connect to MaruServer at tcp://localhost:5555

Verify maru-server is running and accessible.

CXL Memory Exhausted

ERROR: Cannot allocate page for key ...

Increase maru_pool_size or restart maru-server to free memory.

chunk_tokens Alignment Warning

WARNING: maru_kv_chunk_tokens 300 not aligned to block_size 16, adjusted to 288

Normal behavior. Automatically adjusted to a multiple of vLLM’s block_size.

BFloat16 Store Errors

TypeError: Got unsupported ScalarType BFloat16
# or
RuntimeError: can't convert bfloat16 to numpy

BFloat16 models (e.g., Llama 3) fail with numpy-based serialization because numpy has no bfloat16 dtype. The connector handles this by using torch.Tensor.contiguous() and raw byte views instead of .numpy(). If you see this error, ensure you are using the connector from this PR or later — older versions may use numpy conversion paths.

Garbage Output on Second Instance

KV cache data corruption usually means chunks were concatenated as 1D bytes instead of being injected per-chunk. Ensure per-chunk injection is used (the current connector handles this correctly).

For runnable examples, see vLLM Examples.

See also: Architecture Overview, MaruHandler Design, LMCache