vLLM Examples

Runnable examples of the Maru-vLLM direct connector (MaruKVConnector), which moves KV cache through CXL shared memory without LMCache in the serving path. They live under examples/vllm/:

Example	Directory	What it shows
Single-instance verification	examples/vllm/single/	One instance stores KV and reuses it on a repeat request — the cheapest connector check
P2P KV cache sharing	examples/vllm/p2p_sharing/	Two instances share KV cache, so the second skips prefill
CXL on memory nodes over RDMA	examples/vllm/remote_cxl/	Engines on nodes without CXL share KV in a memory node’s CXL pool over RDMA

Start with the single-instance example: it needs one GPU and isolates the store and load paths before any cross-instance behavior is involved.

Prerequisites

• 1 GPU for the single-instance example, 2+ for P2P

• Maru installed with its KV placement kernels: ./install.sh from the Maru source tree, or uv pip install -e /path/to/maru --no-build-isolation in an environment that already has PyTorch and the CUDA toolkit. The connector carries its own CUDA kernels for the coalesced multi-layer transfers and falls back to a transfer per layer without them, which is materially slower. python -c "import maru_kv_ops; print(maru_kv_ops.is_available())" says which path it will take.

• vLLM v0.14+ installed

• maru-server binary available

Single-instance verification

One vLLM instance stores KV to Maru on the first request and loads it back on a repeat. vLLM’s own prefix cache is disabled in the launcher so that Maru is the only cache source — otherwise the repeated prompt would be served from the GPU-resident prefix cache and neither connector path would run.

            vLLM (GPU 0, prefix cache OFF)
                      |
                MaruKVConnector
                      |
                 MaruHandler ── CXL Shared Memory
                      |
                 MaruServer (metadata)

1st request → cold: full prefill, KV stored to Maru
2nd request → warm: prefix found in Maru, KV loaded, prefill skipped

Automated

cd examples/vllm/single
./single_example.sh                              # Default: Qwen/Qwen2.5-0.5B
./single_example.sh --model meta-llama/Llama-3-8B

The script starts maru-server and one vLLM instance, then sends the same prompt twice. Cache Hit: Yes with a TTFT speedup confirms both paths. It exits non-zero on a cache miss; check single.log for connector errors (a missing Maru: loaded N layers message means the worker load path failed).

Step-by-step

cd examples/vllm/single

# 1. Start maru-server
source env.sh
maru-server --port $MARU_SERVER_PORT

# 2. Launch the instance (GPU 0, prefix cache OFF)
./single_vllm_launcher.sh Qwen/Qwen2.5-0.5B

# 3. Verify
./run_simple_query.sh                            # same prompt twice
./run_benchmark.sh --model Qwen/Qwen2.5-0.5B     # TTFT: cold vs warm

Configuration

Settings live in examples/vllm/single/env.sh and can be overridden through the environment:

Variable	Default	Description
MARU_SERVER_PORT	10000 + uid	MaruServer port
MARU_INST_PORT	12000 + uid + 20	vLLM instance port
MARU_POOL_SIZE	4G	CXL shared memory pool size
MARU_KV_CHUNK_TOKENS	256	Tokens per KV cache chunk
GPU_MEM_UTIL	0.1	vLLM GPU memory utilization

P2P KV cache sharing

Two vLLM instances share KV cache through CXL shared memory. The first stores the prefix, the second loads it and skips prefill.

Instance 1 (GPU 0)                    Instance 2 (GPU 1)
     vLLM                                  vLLM
       |                                     |
  MaruKVConnector                      MaruKVConnector
       |                                     |
       +----------- MaruHandler -------------+
                        |
                   CXL Shared Memory
                        |
                   MaruServer (metadata)

Automated

cd examples/vllm/p2p_sharing
./p2p_example.sh                                 # Default: Qwen/Qwen2.5-0.5B
./p2p_example.sh meta-llama/Llama-3-8B

The script starts maru-server, launches both instances, runs the sharing test, and cleans up every process it started.

Step-by-step

cd examples/vllm/p2p_sharing

# 1. Start maru-server
source env.sh
maru-server --port $MARU_SERVER_PORT

# 2. Launch the instances (separate terminals)
./p2p_vllm_launcher.sh inst1 Qwen/Qwen2.5-0.5B   # GPU 0
./p2p_vllm_launcher.sh inst2 Qwen/Qwen2.5-0.5B   # GPU 1

# 3. Run the test
./run_simple_query.sh                            # prompt + output verification
./run_benchmark.sh --model Qwen/Qwen2.5-0.5B     # TTFT measurement

Instance 2 should report a lower TTFT and Cache Hit: Yes, because it loads the KV cache from CXL instead of recomputing prefill.

Configuration

Settings live in examples/vllm/p2p_sharing/env.sh:

Variable	Default	Description
MARU_SERVER_PORT	10000 + uid	MaruServer port
MARU_INST1_PORT	12000 + uid + 10	vLLM instance 1 port
MARU_INST2_PORT	12000 + uid + 11	vLLM instance 2 port
MARU_POOL_SIZE	4G	CXL shared memory pool size
MARU_KV_CHUNK_TOKENS	256	Tokens per KV cache chunk
GPU_MEM_UTIL	0.1	vLLM GPU memory utilization

The two examples use different instance ports, so they do not collide when run on the same machine.

Troubleshooting

Instance 2 TTFT is not faster. Check the Instance 2 log for Maru: loaded N layers. If it is absent, confirm maru-server is running and that both instances connect to it. For very small models with short prompts the CXL retrieve overhead can exceed the prefill it saves.

Garbage output on Instance 2. KV cache corruption — confirm the load is using per-chunk injection rather than a concatenated 1D load.

CXL on memory nodes over RDMA

Engines on nodes without CXL keep KV in the CXL pool of memory nodes and move it over RDMA (maru_storage_backend="remote_cxl"). Each memory node runs a remote CXL agent that maps and registers regions for the workers; maru-server allocates the regions as for local CXL.

cd examples/vllm/remote_cxl
# One host with 2 GPUs, an RDMA NIC, a CXL/DAX device, the resource manager
# running (sudo systemctl start maru-resource-manager) and NIXL (uv pip install nixl)
MARU_DAX_PATH=/dev/dax0.0 MARU_AGENT_UCX_DEVICE=mlx5_0:1 \
MARU_WORKER_UCX_DEVICE=mlx5_0:1 ./remote_cxl_example.sh

Pool hit: Yes and Same answer: Yes confirm that the second instance loaded the first one’s KV from the pool. See examples/vllm/remote_cxl/README.md for running the memory node, maru-server and workers on separate nodes.

Further reading

Each example directory has a README with its full file listing. For connector configuration — including the asynchronous load and store settings — see vLLM.