vllm
Zero-Dependency Dense & Sparse MoE GGUF Inference Engine (SIMD & Vulkan UMA)

vllm — Dense & Sparse Mixture-of-Experts (MoE) Inference Engine

vllm — Dense & Sparse Mixture-of-Experts Inference Engine

Overview

vllm is a zero-dependency, single-binary Dense and Sparse Mixture-of-Experts (MoE) Large Language Model Inference Engine and OpenAI-Compatible Serving Runtime. Distributed as a self-contained native executable for Windows, Linux, and macOS, vllm maps multi-gigabyte GGUF v2/v3 model files directly into virtual memory in milliseconds and executes quantized inference across multi-threaded x86_64 AVX2/FMA/F16C, ARM64 NEON, and Vulkan 1.3 Unified Memory Architecture (UMA) compute shaders.

SERVING & INFERENCE TOPOLOGY

End-to-End Inference & OpenAI-Compatible HTTP / SSE Serving Topology

Click any component node to inspect how client requests traverse tokenization, KV-cached transformer execution, sampling, and UTF-8 streaming.

Interactive SVG • Click any node (7 components)
1. CLIENT & CLI INGRESS SDKs / curl / Interactive CLI 2. PROTOCOL, TOKENIZATION & INFERENCE CORE HTTP/1.1 + ChatML BPE + Stateful Transformer 3. SAMPLING & STREAM DECODE Top-K/Top-P/Min-P + UTF-8 SSE HTTP / JSON Prompt / REPL Messages Token IDs Vocab Logits Next Token API Clients OpenAI SDKs & Agents POST /v1/chat/completions Operator CLI CLI Subcommands run, chat, serve, bench, gpu Protocol Core OpenAI HTTP/SSE Server Zero-Reflection JSON & SSE Tokenization ChatML & BPE Tokenizer GPT-2 Byte-Level + SentencePiece Compute Core Transformer Runtime Zero-Alloc Forward & Batch Prefill Sampling Logit Sampler Chain Repetition -> Top-K -> Top-P -> Min-P Output Stream UTF-8 Stream Decoder Multi-Byte Codepoint Buffer
Components:
API Clients
OpenAI SDKs & Agents
External OpenAI Client Boundary
Component 1 of 7
Architectural Role & Purpose

External applications, IDE assistants, Python/JS OpenAI SDKs, and curl clients consuming local LLM endpoints.

Contextual Description & System Interaction

Connects over loopback (127.0.0.1 by default) or an authenticated API gateway without external cloud dependencies.

Core Functionality & Execution Mechanics
  • Submits synchronous JSON or Server-Sent Events (stream=true) chat requests
  • Queries GET /v1/models and GET /health for readiness and model discovery
  • Sends plain string messages or multimodal OpenAI content-part arrays
Operator CLI
CLI Subcommands
Command-Line Control Surface
Component 2 of 7
Architectural Role & Purpose

Direct terminal workflow entrypoints for single-turn generation, multi-turn REPL chat, GGUF inspection, and GPU probing.

Contextual Description & System Interaction

Shares the exact same zero-allocation inference and tokenizer pipeline used by the HTTP serving daemon.

Core Functionality & Execution Mechanics
  • Executes interactive multi-turn sessions with /reset and /exit commands
  • Runs deterministic prefill/decode throughput benchmarks via bench
  • Automatically fetches missing GGUF artifacts over HTTPS TLS 1.2+ staging
Protocol Core
OpenAI HTTP/SSE Server
OpenAI HTTP & SSE Gateway
Component 3 of 7
Architectural Role & Purpose

Parses bounded HTTP/1.1 requests with a recursive-descent JSON parser and streams incremental SSE chunks.

Contextual Description & System Interaction

Eliminates dynamic reflection overhead while maintaining wire compatibility with OpenAI chat completion schemas.

Core Functionality & Execution Mechanics
  • Enforces 30s read / 60s write socket timeouts, 64 KiB header cap, and 1 MiB body limit
  • Limits JSON nesting depth to 32 levels to prevent stack exhaustion
  • Flushes data: {...} SSE delta frames on every decoded token boundary
Tokenization
ChatML & BPE Tokenizer
ChatML & BPE Tokenizer Subsystem
Component 4 of 7
Architectural Role & Purpose

Formats system/user/assistant turns with injection-safe ChatML markers and encodes text into integer token IDs.

Contextual Description & System Interaction

Guarantees structural separation between trusted conversation framing and untrusted user prompt content.

Core Functionality & Execution Mechanics
  • Encodes untrusted user text via ordinary BPE so control tokens cannot be spoofed
  • Supports GPT-2 256-byte unicode tables (Qwen2/OLMoE) and SentencePiece styles
  • Applies priority-ranked BPE merge pairs directly from GGUF vocabulary arrays
Compute Core
Transformer Runtime
Stateful Transformer Inference Runtime
Component 5 of 7
Architectural Role & Purpose

Executes dense (LLaMA/Qwen2/Mistral/Gemma) and sparse MoE (OLMoE) transformer layers over pre-allocated arenas.

Contextual Description & System Interaction

Coordinates quantized weight views, GQA attention, sparse expert routing, CPU SIMD workers, and Vulkan UMA.

Core Functionality & Execution Mechanics
  • Matches longest shared token prefix in KV cache to skip redundant prefill work
  • Dispatches batched prefill across prompt deltas and single-token autoregressive decode
  • Executes with zero heap allocations per generated token across all layers
Sampling
Logit Sampler Chain
Non-Mutating Logit Sampler Pipeline
Component 6 of 7
Architectural Role & Purpose

Selects the next token ID deterministically (greedy) or stochastically from vocabulary logits without mutating model state.

Contextual Description & System Interaction

Prevents subtle logit-aliasing bugs across multi-turn steps while supporting creative and deterministic decoding.

Core Functionality & Execution Mechanics
  • Copies logits into a private scratch buffer before applying repetition penalties
  • Chains temperature scaling, partial-sort Top-K, nucleus Top-P, and Min-P filtering
  • Drives reproducible sampling via seeded xoshiro PRNG state
Output Stream
UTF-8 Stream Decoder
Incremental UTF-8 Stream Decoder
Component 7 of 7
Architectural Role & Purpose

Reassembles byte-level BPE pieces across token boundaries so partial UTF-8 sequences never corrupt client output.

Contextual Description & System Interaction

Ensures CJK characters, mathematical symbols, and multi-byte sequences stream cleanly over CLI stdout and HTTP SSE.

Core Functionality & Execution Mechanics
  • Buffers incomplete 2-byte, 3-byte, and 4-byte UTF-8 lead/continuation bytes
  • Emits clean UTF-8 strings immediately once codepoints complete
  • Validates EOS and ChatML stop tokens (<|im_end|>, <|endoftext|>) per step

Core Capabilities

Audience Capability Operational Value
Application Developers Drop-In OpenAI REST & SSE Server (serve) Point official OpenAI Python/Node.js SDKs, IDE coding assistants, local RAG pipelines, and curl clients at http://127.0.0.1:8080/v1/chat/completions for synchronous JSON or real-time text/event-stream SSE token streaming.
AI & ML Practitioners Dense + Sparse MoE GGUF Execution (run, chat) Run compact dense GQA models (Qwen2.5, LLaMA 3, Mistral, Gemma, Phi) and sparse 64 -> Top-8 Mixture-of-Experts models (OLMoE-1B-7B) directly from quantized GGUF files (Q4_K_M, Q5_0, Q6_K, Q8_0, BF16, F16, F32) with zero Python or CUDA runtime dependencies.
Multi-Turn Chat Users Automatic Prefix KV-Cache Reuse Consecutive chat turns automatically reuse the longest matching token prefix stored in the compact FP16 KV Cache, prefilling only newly appended user messages for snappy follow-up responses.
Platform & Edge Operators Zero-Copy Virtual Memory + Vulkan 1.3 UMA Cold-start a 4.0 GB MoE model in ~28 ms via read-only OS memory mapping (CreateFileMappingW / mmap) and accelerate large matrix projections on integrated APUs or discrete GPUs without installing a Vulkan SDK.
Security Teams Loopback Isolation & Hardened Binary Guards Default 127.0.0.1 binding, strict GGUF offset/overflow validation, injection-safe ChatML ordinary-token encoding, and bounded HTTP/JSON framing protect air-gapped workstations and enterprise inference nodes.

Live Diagnostics & Serving Previews

Multi-Turn OpenAI Chat & Streaming SSE Session

vllm Live Multi-Turn OpenAI Chat Session with Prefix KV-Cache Reuse

Zero-Copy GGUF v3 Model & Quantized Tensor Inspector

vllm Zero-Copy GGUF v3 Model & Tensor Catalog Inspector

Dynamic Vulkan 1.3 Unified Memory & Wave64 Shader Probe

vllm Dynamic Vulkan 1.3 Unified Memory & SPIR-V Shader Probe


End-User Documentation Suite

  1. Overview & Readme — Product overview, supported model families, live previews, and quickstart commands.
  2. How-To Guide — Step-by-step recipes for inspecting GGUF models, running interactive chat, serving OpenAI SSE endpoints, enabling Vulkan GPU compute, and connecting SDKs.
  3. End-User Guide — Prompting workflows, multi-turn ChatML sessions, sampling controls (--temp, --top-k, --top-p, --min-p, --repeat-penalty), and choosing between Dense and Sparse MoE models.
  4. Administrator Guide — Production installation, memory & KV-cache sizing, reverse-proxy TLS and Bearer token authentication, service management, and air-gapped model staging.
  5. Operations, Use Cases & Examples — Real-world deployment topologies and runnable curl, Python OpenAI SDK, and JavaScript/TypeScript streaming client examples.
  6. Configuration & CLI Reference — Complete reference for all CLI subcommands, runtime flags, GGUF metadata keys, and HTTP/SSE wire schemas.
  7. Performance & Capacity Guide — Measured SIMD (AVX2/FMA/NEON) and Vulkan Wave64 UMA throughput, Q4_K/Q8_0/BF16 memory sizing formulas, and GQA KV-cache prefix reuse benchmarks.
  8. Consolidated User Handbook — Single-volume master compendium combining all End-User, How-To, Administrator, Operations, Configuration, and Performance guides.

Quickstart

Launch vllm using the prebuilt binary for your platform (dist/bin/windows/vllm.exe, dist/bin/linux/vllm, or dist/bin/macos/vllm):

BASH
# 1. Probe Vulkan 1.3 GPU / APU unified memory heaps and Wave64 SPIR-V shader latency
./vllm gpu

# 2. Inspect GGUF v2/v3 architecture hyperparameters and quantized tensor distribution
./vllm inspect models/olmoe-1b-7b-0924-instruct-q4_k_m.gguf

# 3. Generate a single-turn instruction response using ChatML formatting
./vllm run models/qwen2.5-0.5b-instruct-q4_k_m.gguf --chat -p "Explain zero-copy memory mapping in two sentences." -n 96

# 4. Launch an interactive multi-turn terminal chat session
./vllm chat models/olmoe-1b-7b-0924-instruct-q4_k_m.gguf --system "You are a concise systems architect."

# 5. Start the OpenAI-compatible HTTP & SSE server on loopback port 8080
./vllm serve models/qwen2.5-0.5b-instruct-q4_k_m.gguf --host 127.0.0.1 --port 8080 --ctx 4096

vllm — Step-by-Step How-To Guide

vllm — Dense & Sparse Mixture-of-Experts Inference Engine

Overview

This guide provides task-oriented recipes for End Users, Application Developers, and System Administrators operating vllm across local workstations, integrated APU systems, and private inference servers.


Recipe 1: Inspect a GGUF Model Before Deployment

Before loading an unfamiliar GGUF file into production memory, inspect its architecture metadata, context ceiling, attention head layout, expert routing configuration, and quantization distribution.

  1. Run the inspect subcommand against any .gguf v2 or v3 artifact:
BASH
./vllm inspect models/olmoe-1b-7b-0924-instruct-q4_k_m.gguf
  1. Review the structured summary emitted in milliseconds via zero-copy OS memory mapping:
TEXT
Mapped 4018.32 MB in 28 ms (GGUF v3, 195 tensors, 35 metadata KV pairs)
=== Model Summary ===
  Architecture : olmoe (AllenAI OLMoE 1B 7B 0924 Instruct)
  Layers       : 16
  Hidden Dim   : 2048
  Attn Heads   : 16 Q / 16 KV (head_dim=128, rope_theta=10000.0)
  Context Len  : 4096
  Vocab Size   : 50304
  MoE Experts  : 8 active / 64 total (moe_inter_dim=1024, shared=0)
  Data Offset  : 0x5c3720 (alignment=32 bytes)
  Tensor Types : {'q4_k': 97, 'f32': 81, 'q6_k': 17}
  1. Verify that Context Len, Attn Heads, and Tensor Types match your host RAM budget (see the Performance & Capacity Guide for exact sizing formulas).

Recipe 2: Run Deterministic vs Creative Single-Turn Prompts

Use vllm run for scripted CLI pipelines, automated evaluations, and shell one-liners.

  1. Deterministic Greedy Decoding (--temp 0) — ideal for code generation, structured extraction, and regression testing:
BASH
./vllm run models/qwen2.5-0.5b-instruct-q4_k_m.gguf \
  --chat \
  --system "Reply with valid JSON only." \
  -p "List three advantages of FP16 KV caching as a JSON array of strings." \
  --temp 0 \
  -n 128
  1. Reproducible Stochastic Sampling (--seed + --top-k + --top-p + --min-p) — ideal for natural prose while maintaining exact reproducibility across runs:
BASH
./vllm run models/olmoe-1b-7b-0924-instruct-q4_k_m.gguf \
  --chat \
  -p "Summarize why Sparse Mixture-of-Experts models achieve 7B quality at 1B compute cost." \
  --temp 0.6 \
  --top-k 40 \
  --top-p 0.90 \
  --min-p 0.05 \
  --repeat-penalty 1.10 \
  --seed 42 \
  -n 160

Recipe 3: Conduct Multi-Turn Interactive REPL Sessions

Use vllm chat to hold stateful multi-turn conversations in your terminal with incremental KV-cache reuse across turns.

  1. Start an interactive session with a custom system prompt and context window:
BASH
./vllm chat models/qwen2.5-0.5b-instruct-q4_k_m.gguf \
  --ctx 4096 \
  --temp 0.4 \
  --system "You are a concise Linux and network engineering assistant."
  1. Enter questions at the >>> prompt. Notice that follow-up turns prefill only the newly added user message because prior conversation tokens remain resident in the FP16 KV cache.
  2. Use built-in session commands at any time:
    • Type /reset to clear conversation history and reset KV cache position to 0.
    • Type /exit or /quit (or press Ctrl+D) to close the session cleanly.

Recipe 4: Probe & Enable Vulkan 1.3 Unified Memory GPU Acceleration

On laptops, mini-PCs with integrated APUs (such as AMD Radeon 890M / Ryzen AI), or discrete Vulkan GPUs, verify driver readiness before enabling --device gpu.

  1. Run the built-in Vulkan compute probe:
BASH
./vllm gpu
  1. Confirm that vllm detects your Vulkan 1.2/1.3 device, lists available memory heaps, and passes the [2048 x 2048] Wave64 FP32 and FP16 matrix-vector verification benchmark (y[0]=767.06).
  2. Enable GPU acceleration on run, chat, bench, or serve with --device gpu:
BASH
./vllm run models/qwen2.5-0.5b-instruct-q4_k_m.gguf \
  --device gpu \
  --chat \
  -p "Explain unified memory architecture on modern APUs." \
  -n 128

Recipe 5: Serve OpenAI-Compatible Endpoints for Local SDKs & IDE Tools

Expose any local GGUF model over the standard OpenAI /v1/chat/completions HTTP and Server-Sent Events (SSE) protocol.

  1. Start the server bound to loopback (127.0.0.1) on port 18510:
BASH
./vllm serve models/qwen2.5-0.5b-instruct-q4_k_m.gguf \
  --host 127.0.0.1 \
  --port 18510 \
  --ctx 4096 \
  --threads 12
  1. Verify server health and model catalog readiness:
BASH
curl -s http://127.0.0.1:18510/health
curl -s http://127.0.0.1:18510/v1/models
  1. Stream real-time SSE token deltas with curl -N:
BASH
curl -N http://127.0.0.1:18510/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-0.5b-instruct-q4_k_m.gguf",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "Give three tips for low-latency LLM serving."}
    ],
    "temperature": 0.3,
    "max_tokens": 120,
    "stream": true
  }'

vllm Live OpenAI Wire Protocol & SSE Frame Inspector


Recipe 6: Benchmark Prefill and Decode Throughput

Measure repeatable prompt-prefill (tok/s) and token-decode (tok/s) performance on your hardware before sizing production capacity.

BASH
./vllm bench models/qwen2.5-0.5b-instruct-q4_k_m.gguf -n 128 --threads 12
./vllm bench models/olmoe-1b-7b-0924-instruct-q4_k_m.gguf -n 64 --threads 12

Compare CPU thread counts (--threads 4, --threads 8, --threads 12) against your CPU's physical core count; memory-bandwidth-bound quantized inference typically peaks at the number of physical performance cores rather than hyperthreaded logical threads.

vllm — End-User Guide

vllm — Dense & Sparse Mixture-of-Experts Inference Engine

1. Welcome to Local Private Inference

vllm lets you run modern instruction-tuned and base Large Language Models completely offline on your desktop, laptop, or private workstation. Prompts, documents, source code, and generated responses never leave your machine.

Whether you interact through the terminal (vllm run and vllm chat) or connect your favorite OpenAI-compatible desktop chat client, IDE extension, or local automation script to vllm serve, this guide explains how to choose models, shape prompts, and tune generation behavior.


2. Choosing Between Dense and Sparse MoE Models

vllm supports both Dense Transformer families and Sparse Mixture-of-Experts (MoE) architectures out of the box:

Model Family Example GGUF Artifact Disk / RAM Size Best Suited For
Compact Dense GQA (Qwen2.5-0.5B .. 3B) qwen2.5-0.5b-instruct-q4_k_m.gguf ~469 MB (< 800 MB RSS) Instantaneous coding completions, classification, JSON formatting, edge devices, and rapid multi-turn drafting at 20–30+ tok/s on CPU.
Mid-Size Dense GQA (LLaMA 3 8B, Qwen2.5-7B, Mistral 7B) qwen2.5-7b-instruct-q4_k_m.gguf ~4.4 GB (~5.2 GB RSS) Deep reasoning, long-context summarization, multilingual translation, and complex multi-step analysis on 8–16 GB systems.
Sparse Mixture-of-Experts (OLMoE-1B-7B) olmoe-1b-7b-0924-instruct-q4_k_m.gguf ~4.0 GB (~4.5 GB RSS) 7B-class knowledge capacity at ~1.3B active compute per token — routes each token through only 8 of 64 experts, cutting CPU/RAM bandwidth by 87.5% relative to a dense 7B FFN.

vllm Sparse Mixture-of-Experts 64-to-Top-8 Routing Telemetry


3. Working with Interactive Chat (vllm chat)

Interactive mode wraps every exchange in clean multi-turn conversation framing and keeps your conversation context hot in memory:

BASH
./vllm chat models/olmoe-1b-7b-0924-instruct-q4_k_m.gguf \
  --system "Explain concepts clearly using concrete examples." \
  --temp 0.6 \
  --top-p 0.90

Session Lifecycle & Prefix Caching

  1. First Turn: vllm encodes the system instruction and your first user message, prefills those tokens in a parallel batch, and streams the assistant's reply word by word.
  2. Follow-Up Turns: When you ask a follow-up question, vllm detects that the earlier turns are already stored in its FP16 KV Cache and computes only the new user message before streaming the next reply.
  3. Starting Fresh (/reset): When switching topics, type /reset at the >>> prompt to clear the conversation buffer and reclaim the full --ctx window without restarting or re-mapping the model file.
  4. Exiting (/exit or /quit): Ends the session cleanly and releases memory mappings.

4. Understanding Sampling Controls

Tuning sampling flags changes how vllm chooses each token from the model's probability distribution:

Parameter CLI Flag / JSON Field Recommended Range Effect on Output
Temperature --temp / temperature 0.0 – 1.2 (default 0.7) 0.0 selects the single most probable token at every step (deterministic greedy). 0.2–0.4 favors factual, focused answers; 0.7–0.9 encourages varied phrasing and brainstorming.
Top-K --top-k / top_k 1 – 100 (default 40) Restricts sampling to the K highest-probability candidate tokens, preventing low-probability tail noise. Set 0 to disable.
Top-P (Nucleus) --top-p / top_p 0.50 – 1.00 (default 0.90) Dynamically keeps the smallest set of top tokens whose cumulative probability reaches P, adapting candidate breadth to model confidence.
Min-P --min-p / min_p 0.00 – 0.20 (default 0.05 CLI) Filters out any candidate token whose probability is less than min_p * max_prob, maintaining coherence even at higher temperatures.
Repetition Penalty --repeat-penalty / repetition_penalty 1.00 – 1.20 (default 1.10 CLI) Gently down-weights tokens that have already appeared in the conversation (1.0 disables the penalty).
Random Seed --seed / seed Any 64-bit integer (0 = time seed) Setting a fixed non-zero seed (for example --seed 42) produces bit-for-bit repeatable generations for identical prompts and flags.

Quick Preset Profiles

  • Code Generation, SQL & JSON Extraction: --temp 0 --repeat-penalty 1.0
  • Technical Q&A & Summarization: --temp 0.3 --top-k 40 --top-p 0.90 --min-p 0.05
  • Creative Writing & Brainstorming: --temp 0.8 --top-k 60 --top-p 0.95 --min-p 0.05 --repeat-penalty 1.12

5. Connecting Desktop GUIs & Coding Assistants

When vllm serve is running locally, any tool that supports a custom OpenAI Base URL can use your local model:

  1. Start vllm serve:
BASH
./vllm serve models/qwen2.5-0.5b-instruct-q4_k_m.gguf --host 127.0.0.1 --port 8080 --ctx 4096
  1. Configure your client application with:
    • API Base URL: http://127.0.0.1:8080/v1
    • API Key: local-vllm (any non-empty placeholder string accepted by your client when connecting directly over loopback)
    • Model Name: qwen2.5-0.5b-instruct-q4_k_m.gguf (or query http://127.0.0.1:8080/v1/models to auto-populate)

Both standard single-string messages ("content": "Hello") and modern OpenAI array-of-parts messages ("content": [{"type": "text", "text": "Hello"}]) sent by newer agent frameworks and IDE plugins are supported transparently.

vllm — System Administrator & Deployment Guide

vllm — Dense & Sparse Mixture-of-Experts Inference Engine

1. Deployment Architecture Overview

vllm is engineered for zero-runtime-dependency operations:

  • Single Static/Native Binary: No Python virtual environments, PyTorch wheels, CUDA toolkit installations, or C++ shared-library version conflicts.
  • Read-Only Memory-Mapped Weights: Model .gguf files are opened strictly read-only (PAGE_READONLY on Windows, PROT_READ + MAP_PRIVATE on Linux/macOS). Multiple processes can map the same physical RAM pages safely, and power loss can never corrupt model weights.
  • Deterministic Arena Memory: All activation buffers, quantized scratchpads, and FP16 KV caches are pre-allocated at startup for --ctx <N>. During steady-state token generation, vllm performs zero per-token heap allocations, eliminating out-of-memory spikes mid-request.
  • Loopback-First Network Security: vllm serve binds 127.0.0.1 by default so workstation deployments never inadvertently expose an unauthenticated port on Wi-Fi or corporate LAN interfaces.

2. Platform Installation

Windows x64

  1. Extract vllm-windows-x64.zip into an installation directory such as ./vllm/ (your extracted product root directory).
  2. Verify archive integrity against SHA256SUMS.txt:
POWERSHELL
Get-FileHash .\vllm-windows-x64.zip -Algorithm SHA256
  1. Optionally verify Vulkan 1.3 integrated/discrete GPU availability:
POWERSHELL
.\bin\vllm.exe gpu

Linux x86_64

  1. Verify checksum and extract vllm-linux-x64.tar.gz:
BASH
sha256sum -c SHA256SUMS.txt --ignore-missing
mkdir -p ./bin ./models
tar -xzf vllm-linux-x64.tar.gz --strip-components=1
chmod 0755 ./bin/vllm
  1. Create a dedicated unprivileged service account:
BASH
sudo useradd --system --home-dir . --shell nologin vllm
sudo chown -R root:vllm ./models
sudo chmod 0750 ./models
sudo chmod 0440 ./models/*.gguf

macOS (Apple Silicon ARM64 & Intel x86_64)

  1. Extract vllm-macos-src-bundle.tar.gz (cross-compiled standalone C bundle) or vllm-macos-native.tar.gz (when built natively on macOS):
BASH
tar -xzf vllm-macos-src-bundle.tar.gz
cd vllm-macos-src-bundle/bin
  1. If compiling from the standalone macOS C bundle on a target Mac with Xcode Command Line Tools (clang), execute the included helper script:
BASH
chmod +x build-macos.sh
./build-macos.sh
./vllm gpu

3. Air-Gapped & Verified Model Staging

In regulated or air-gapped environments, administrators should pre-stage verified GGUF artifacts rather than relying on on-demand internet downloads:

  1. Download the approved .gguf artifact on a staging bastion over HTTPS (TLS 1.2+).
  2. Record and verify its SHA-256 digest against your internal model registry manifest.
  3. Run ./vllm inspect <path-to-model.gguf> on the staging host to verify header integrity, tensor bounds, and expected quantization layout before copying the file into ./models/.
  4. Mount ./models read-only (ro,nosuid,nodev,noexec) inside containers or systemd units.

4. Hardened Linux systemd Service Configuration

Run vllm serve on loopback under strict systemd sandboxing:

INI
# ./deploy/vllm.service
[Unit]
Description=vllm Zero-Dependency GGUF Inference Server
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=vllm
Group=vllm
WorkingDirectory=.
ExecStart=./bin/vllm serve ./models/olmoe-1b-7b-0924-instruct-q4_k_m.gguf \
  --host 127.0.0.1 \
  --port 18510 \
  --ctx 4096 \
  --threads 8
Restart=on-failure
RestartSec=3

# Hardening Directives
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadOnlyPaths=./models
CapabilityBoundingSet=
LockPersonality=true
RestrictSUIDSGID=true

[Install]
WantedBy=multi-user.target

Enable and verify the unit:

BASH
sudo systemctl daemon-reload
sudo systemctl enable --now vllm.service
curl -fsS http://127.0.0.1:18510/health

5. Reverse-Proxy TLS Termination & API Bearer Authentication

vllm serve intentionally keeps its HTTP parser minimal and binds 127.0.0.1 by default. When exposing vllm across a network segment or team VLAN, place an identity-aware gateway (vgate) or TLS reverse proxy (Caddy / NGINX) in front of 127.0.0.1:18510 to enforce TLS 1.3, Bearer API Key / OIDC JWT authentication, and SSE flush-through (proxy_buffering off).

Example NGINX TLS + Bearer Token Reverse Proxy

NGINX
map $http_authorization $vllm_auth_ok {
    default 0;
    "Bearer YOUR_ROTATED_HIGH_ENTROPY_API_TOKEN" 1;
}

server {
    listen 443 ssl http2;
    server_name llm-internal.corp.example;

    ssl_certificate     ./certs/llm-internal.pem;
    ssl_certificate_key ./certs/llm-internal.key;
    ssl_protocols       TLSv1.2 TLSv1.3;

    client_max_body_size 1m;

    location / {
        if ($vllm_auth_ok = 0) {
            return 401 '{"error":{"message":"Unauthorized API token","type":"auth_error"}}';
        }
        proxy_pass http://127.0.0.1:18510;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_buffering off;
        proxy_cache off;
        proxy_read_timeout 120s;
    }
}

vllm Runtime Security Guardrails & Bounds Enforcement


6. Health Monitoring, Capacity Controls & Troubleshooting

Symptom / Check Diagnostic Command / Cause Resolution
Readiness Probe curl -fsS http://127.0.0.1:18510/health Expect HTTP 200 with {"status":"ok","model":"...","ready":true} (lock-free; never blocked by active inference). Wire this into load balancer or systemd watchdogs.
Prometheus Metrics curl -fsS http://127.0.0.1:18510/metrics Exposes vllm_requests_total, vllm_requests_active, vllm_errors_total, vllm_tokens_generated_total, vllm_kv_prefix_hits_total, and vllm_inference_duration_ms_sum.
HTTP 413 Payload Too Large / HTTP 431 Request body exceeds 1 MiB or headers exceed 64 KiB Trim oversized RAG document chunks before sending to /v1/chat/completions; vllm enforces strict 1 MiB body and 64 KiB header bounds.
Lower-Than-Expected CPU tok/s --threads set equal to logical hyperthreads instead of physical cores Set --threads equal to physical performance cores (e.g. 8 or 12) to prevent L3 cache and DRAM bus contention.
No Vulkan-capable GPU detected Headless VM or missing vendor ICD (vulkan-1.dll / libvulkan.so.1) Install the host GPU driver's Vulkan runtime or omit --device gpu to run on the AVX2/NEON CPU thread pool.
Long Multi-Turn Context Truncation Conversation exceeds --ctx (default 2048 if overridden below model limit) Increase --ctx 4096 or --ctx 8192 at server startup and verify host RAM headroom for the expanded FP16 KV arena.

vllm — Operations, Use Cases & Client Examples

vllm — Dense & Sparse Mixture-of-Experts Inference Engine

1. Operational Deployment Patterns

vllm fits four common enterprise and developer deployment patterns without requiring dedicated datacenter GPU clusters:

Topology Target Hardware Model Selection Operational Pattern
1. Developer Loopback Assistant Laptop / Workstation (16–32 GB RAM, AVX2 or Apple Silicon NEON) qwen2.5-0.5b-instruct-q4_k_m.gguf or olmoe-1b-7b-0924-instruct-q4_k_m.gguf vllm serve bound to 127.0.0.1:8080 backing IDE autocomplete, commit message drafting, and local CLI scripts.
2. Integrated APU Unified-Memory Node Mini-PC / Edge Workstation with Vulkan 1.3 UMA APU (16–64 GB shared RAM) Q4_K_M / Q6_K Dense or Sparse MoE models --device gpu uploads resident projections into host-visible coherent UMA memory with zero PCIe transfer overhead.
3. Local RAG & Knowledge Synthesis Sidecar Co-located Linux service alongside vkbm / vstore Instruction-tuned Q4_K_M or Q8_0 GGUF Synthesizes grounded answers over retrieved knowledge chunks strictly over loopback HTTP (127.0.0.1:18510) with zero external data egress.
4. Team Gateway Behind vgate / NGINX 8–16 physical core server (32–64 GB RAM) OLMoE-1B-7B (Q4_K_M) or 7B–8B Dense GQA Reverse proxy terminates TLS 1.3 and validates OIDC / Bearer tokens before forwarding /v1/chat/completions over loopback.

2. Runnable Integration Examples

Start a local vllm serve instance before running the client examples below:

BASH
./vllm serve models/qwen2.5-0.5b-instruct-q4_k_m.gguf --host 127.0.0.1 --port 18510 --ctx 4096

vllm Live Multi-Turn Session & Prefix KV-Cache Telemetry


Example A: curl Synchronous JSON & Streaming SSE

1. Non-Streaming Completion (stream: false)

BASH
curl -s http://127.0.0.1:18510/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-0.5b-instruct-q4_k_m.gguf",
    "messages": [
      {"role": "system", "content": "You are a concise systems engineering assistant."},
      {"role": "user", "content": "Why does zero-copy memory mapping accelerate GGUF startup?"}
    ],
    "max_tokens": 96,
    "temperature": 0.2,
    "seed": 42,
    "stream": false
  }'

2. Streaming Server-Sent Events (stream: true)

BASH
curl -N http://127.0.0.1:18510/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-0.5b-instruct-q4_k_m.gguf",
    "messages": [
      {"role": "system", "content": "You are a concise systems engineering assistant."},
      {"role": "user", "content": "Explain grouped-query attention in three bullet points."}
    ],
    "max_tokens": 128,
    "temperature": 0.3,
    "stream": true
  }'

Example B: Official Python openai SDK (Multi-Turn Prefix Reuse)

Because vllm implements the standard /v1/chat/completions and /v1/models wire contracts, the official openai Python client works with zero code changes beyond setting base_url:

PYTHON
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:18510/v1",
    api_key="local-vllm",
)

messages = [
    {"role": "system", "content": "You are a concise infrastructure reliability engineer."},
    {"role": "user", "content": "What are two benefits of read-only memory-mapped model weights?"},
]

# Turn 1: Synchronous JSON response
turn1 = client.chat.completions.create(
    model="qwen2.5-0.5b-instruct-q4_k_m.gguf",
    messages=messages,
    temperature=0.2,
    max_tokens=96,
)
reply1 = turn1.choices[0].message.content
print("Turn 1:", reply1)

# Turn 2: Append history and stream SSE deltas (automatically hits vllm's prefix KV cache!)
messages.append({"role": "assistant", "content": reply1})
messages.append({"role": "user", "content": "How does FP16 KV caching complement that?"})

stream = client.chat.completions.create(
    model="qwen2.5-0.5b-instruct-q4_k_m.gguf",
    messages=messages,
    temperature=0.2,
    max_tokens=96,
    stream=True,
)

print("Turn 2: ", end="", flush=True)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)
print()

Example C: Node.js / TypeScript Zero-Dependency fetch SSE Reader

Modern Node.js (v20+) and browser runtimes can consume vllm's text/event-stream responses directly using native fetch:

JAVASCRIPT
async function streamChatCompletion() {
  const response = await fetch("http://127.0.0.1:18510/v1/chat/completions", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      model: "qwen2.5-0.5b-instruct-q4_k_m.gguf",
      messages: [
        { role: "system", content: "Answer in plain, direct engineering prose." },
        { role: "user", content: "Summarize Sparse Mixture-of-Experts routing." },
      ],
      temperature: 0.3,
      max_tokens: 100,
      stream: true,
    }),
  });

  if (!response.ok) {
    throw new Error(`HTTP ${response.status}: ${await response.text()}`);
  }

  const decoder = new TextDecoder();
  let buffer = "";
  for await (const chunk of response.body) {
    buffer += decoder.decode(chunk, { stream: true });
    const events = buffer.split("\n\n");
    buffer = events.pop() || "";
    for (const evt of events) {
      const line = evt.trim();
      if (!line.startsWith("data: ") || line === "data: [DONE]") continue;
      const payload = JSON.parse(line.slice(6));
      const tokenText = payload.choices?.[0]?.delta?.content ?? "";
      process.stdout.write(tokenText);
    }
  }
  process.stdout.write("\n");
}

streamChatCompletion();

Example D: Air-Gapped Log Triage & Batch CLI Pipeline

For offline batch classification or security log summarization where no network daemon is needed, invoke vllm run directly inside shell pipelines:

BASH
./vllm run models/olmoe-1b-7b-0924-instruct-q4_k_m.gguf \
  --chat \
  --system "Classify the severity (INFO, WARN, CRITICAL) and summarize root cause in one line." \
  -p "2026-11-14T07:12:04Z kernel: nvme0n1: I/O error, dev nvme0n1, sector 8841920 op 0x0:(READ)" \
  --temp 0 \
  -n 64

3. Operational Runbook Checklist

  1. Pre-Flight Verification:
    • Run ./vllm inspect <model.gguf> to confirm GGUF v2/v3 header validity and expected tensor count.
    • Run ./vllm gpu if deploying with --device gpu to verify Vulkan driver initialization and numerical parity.
  2. Steady-State Operations:
    • Monitor GET /health (HTTP 200 OK), GET /metrics (Prometheus counters), and structured JSON request logs on stdout (ts, req_id, status, prompt_tokens, kv_hit, completion_tokens, duration_ms).
    • High kv_hit counts on multi-turn agent workloads confirm that clients are preserving deterministic system prompts at the start of messages, maximizing prefix KV-cache reuse.
  3. Graceful Upgrade:
    • Stage the new vllm binary alongside the existing release, verify SHA256SUMS.txt, drain active HTTP connections at the reverse proxy, and restart the service; read-only .gguf files require zero migration or conversion across binary upgrades.

vllm — Configuration & CLI Reference

vllm — Dense & Sparse Mixture-of-Experts Inference Engine

1. Command-Line Synopsis

TEXT
vllm <command> [model.gguf] [options]

If [model.gguf] is omitted on inspect, run, chat, serve, or bench, vllm defaults to models/olmoe-1b-7b-0924-instruct-q4_k_m.gguf and automatically downloads it over HTTPS (TLS 1.2+) if not already present on disk.


2. Subcommands

Subcommand Arguments Description
download <preset-id> [--sha256 <hex>] Downloads an official OLMoE or Qwen GGUF preset over HTTPS (TLS 1.2+) into ./models/, validates GGUF header/tensor bounds and optional SHA-256 digest before atomic rename.
inspect <model.gguf> Memory-maps the GGUF v2/v3 file read-only, validates all metadata and tensor bounds, and prints architecture hyperparameters, expert counts, data alignment, and per-DType tensor counts.
gpu (none) Dynamically probes the system's Vulkan 1.2/1.3 driver, prints GPU/APU memory heaps, verifies numerical parity (y[0]=767.06), and benchmarks [2048 x 2048] Wave64 FP32 and FP16 SPIR-V matrix-vector dispatches.
run <model.gguf> [options] Executes a single prompt prefill + autoregressive token generation pass and prints prefill/decode throughput (tok/s) upon completion.
chat <model.gguf> [options] Launches an interactive multi-turn ChatML REPL session with automatic prefix KV-cache reuse and /clear, /exit, /quit session commands.
serve <model.gguf> [options] Starts the concurrent OpenAI-compatible HTTP/1.1 & Server-Sent Events (SSE) server exposing /health, /metrics, /v1/models, and /v1/chat/completions with graceful SIGINT/SIGTERM shutdown.
help [-h | --help | -v | --version] Prints command usage syntax, supported model presets, or version metadata (vllm 0.1.0).

3. Complete Runtime Flags Reference

Flag Alias Value Type Default Applicable Commands Description
--prompt -p string "Hello" run Input prompt text to complete (or initial user message when combined with --chat).
--chat — (boolean) false (run), true (chat) run Wraps --system and --prompt in injection-safe ChatML <|im_start|> / <|im_end|> conversation framing.
--system -s string "" run --chat, chat System instruction placed at the head of the ChatML conversation context.
--max-tokens -n int (>= 1) 128 run, chat, serve Maximum number of completion tokens generated per turn or request.
--ctx -c int (>= 16) 2048 (clamped to <= model.context_length) run, chat, serve Maximum KV-cache sequence length pre-allocated at startup.
--threads — int (1..256) Auto-detected physical core count run, chat, serve Number of persistent worker threads in the lock-free SIMD work-stealing thread pool.
--device — cpu | gpu cpu run, chat, serve Selects multi-threaded CPU SIMD execution (cpu) or Vulkan UMA/VRAM GPU offload (gpu) with automatic CPU fallback.
--gpu-index — int (>= 0) Auto (prefers hardware GPU) run, chat, serve Explicit Vulkan physical device index (also configurable via VLLM_GPU_INDEX).
--temp -t f32 (>= 0.0) 0.7 run, chat, serve Sampling temperature (0.0 switches to deterministic argmax greedy decoding).
--top-k — int (>= 0) 40 run, chat, serve Top-K candidate cutoff before nucleus filtering (0 disables Top-K).
--top-p — f32 ((0.0, 1.0]) 0.90 run, chat, serve Nucleus cumulative probability threshold (1.0 disables Top-P).
--min-p — f32 ([0.0, 1.0]) 0.0 run, chat, serve Minimum candidate probability relative to the top token probability (0.0 disables Min-P).
--repeat-penalty — f32 (> 0.0) 1.0 run, chat, serve Multiplicative penalty applied to tokens already present in the last 64 tokens (1.0 disables).
--seed — u64 0 (wall-clock seed) run, chat, serve PRNG seed for deterministic stochastic sampling when non-zero.
--host — string "127.0.0.1" serve IPv4 bind address for the OpenAI HTTP server. Defaults to loopback 127.0.0.1 for local isolation.
--port — int (1..65535) 8080 serve TCP port for the OpenAI HTTP & SSE server.
--api-key — string "" (or VLLM_API_KEY env) serve Optional constant-time Bearer API key required for /v1/* and /metrics endpoints.
--cors-origin — string "" (disabled) serve Opt-in Access-Control-Allow-Origin header value. Disabled by default to prevent localhost CSRF.
--no-prefix-cache — (boolean) false serve Disables cross-request KV prefix cache reuse for multi-tenant isolation.
--max-connections — int (1..4096) 64 serve Maximum concurrent TCP connections accepted by serve (excess connections receive HTTP 503).
--sha256 — string (64 hex) "" download Verifies cryptographic SHA-256 digest of the downloaded .gguf artifact before atomic rename.

4. OpenAI HTTP & SSE API Reference (vllm serve)

4.1 Endpoints Summary

Method Path Response Content-Type Description
GET /health application/json Lock-free liveness and readiness probe ({"status":"ok","model":"<id>","ready":true} or HTTP 503 during graceful shutdown).
GET /metrics text/plain; version=0.0.4 Lock-free Prometheus metrics (vllm_requests_total, vllm_requests_active, vllm_errors_total, vllm_tokens_generated_total, vllm_kv_prefix_hits_total, vllm_inference_duration_ms_sum).
GET /v1/models application/json OpenAI model list catalog returning the active GGUF model ID and owned_by: "vllm".
POST /v1/chat/completions application/json or text/event-stream OpenAI chat completion endpoint supporting multi-turn messages, stop sequences, Transfer-Encoding: chunked, Expect: 100-continue, prefix KV-cache reuse, and stream: true SSE deltas (including stream_options.include_usage).
OPTIONS * application/json Preflight handler; emits CORS headers only when --cors-origin is explicitly configured.

4.2 POST /v1/chat/completions Request Schema

JSON Field Type Default Description
model string Loaded model ID Optional model identifier string echoed in responses.
messages array<object> (required, non-empty) Ordered array of {"role": "system"|"user"|"assistant", "content": "..." | [{"type":"text","text":"..."}]} turns. Supports full Unicode including UTF-16 surrogate pairs.
stream boolean false When true, streams incremental data: {...}\n\n SSE chunks terminated by data: [DONE]\n\n.
stream_options object {} Supports {"include_usage": true} to emit a final usage SSE chunk before data: [DONE].
stop string | array<string> [] Up to 16 stop sequences that halt generation cleanly with finish_reason: "stop".
max_tokens integer Server --max-tokens (max_completion_tokens alias supported) Maximum generated tokens (>= 1, automatically clamped to remaining --ctx headroom).
temperature number 0.7 Sampling temperature in [0.0, 100.0] (0.0 = greedy argmax).
top_p number 0.90 Nucleus sampling threshold in (0.0, 1.0].
top_k integer 40 Top-K candidate cutoff (>= 0).
min_p number 0.00 Relative Min-P probability cutoff in [0.0, 1.0].
repeat_penalty number 1.00 Token repetition penalty multiplier in (0.0, 100.0].
seed integer 0 Deterministic PRNG seed (> 0).

4.3 Protocol & Security Limits

Limit Enforced Value Behavior When Exceeded
Total Read / Write Deadlines 30 s total wall-clock read / 60 s write Closes stalled or Slowloris TCP connections cleanly (HTTP 408 Request Timeout).
Max HTTP Request Header 64 KiB (65,536 bytes) Rejects oversized header blocks with HTTP 431 Request Header Fields Too Large.
Max HTTP Request Body 1 MiB (1,048,576 bytes) Rejects oversized Content-Length or chunked streams with HTTP 413 Payload Too Large.
HTTP Smuggling Guard RFC 9112 §6.3 Rejects conflicting duplicate Content-Length or simultaneous Content-Length + Transfer-Encoding with HTTP 400.
Max Concurrent Connections 64 (configurable via --max-connections) Rejects excess connections cleanly with HTTP 503 Service Unavailable.
Max JSON Nesting Depth 32 levels Aborts recursive JSON parsing cleanly with HTTP 400 to prevent stack exhaustion.

5. Self-Describing GGUF Metadata Keys

vllm requires zero external configuration files because all model hyperparameters are resolved directly from standardized GGUF v2/v3 metadata keys at load time:

Metadata Key Pattern (<arch> = llama, qwen2, olmoe, etc.) Resolved Parameter
general.architecture, general.name, general.alignment Model family dispatch, human-readable name, and tensor data byte alignment (32 bytes default).
<arch>.block_count, <arch>.embedding_length, <arch>.context_length Transformer layer count, hidden dimension d_model, and maximum training context window.
<arch>.attention.head_count, <arch>.attention.head_count_kv, <arch>.attention.key_length Query head count, Grouped-Query Attention (GQA) KV head count, and per-head dimension head_dim.
<arch>.rope.freq_base, <arch>.attention.layer_norm_rms_epsilon Rotary Positional Embedding base frequency (theta) and RMSNorm epsilon (1e-5 / 1e-6).
<arch>.expert_count, <arch>.expert_used_count, <arch>.expert_feed_forward_length Sparse MoE total routed experts (64), active Top-K experts per token (8), and expert FFN intermediate width.
tokenizer.ggml.model, tokenizer.ggml.tokens, tokenizer.ggml.merges, tokenizer.ggml.bos_token_id, tokenizer.ggml.eos_token_id Vocabulary style (gpt2 byte-level or llama SentencePiece), token strings, BPE merge priorities, and BOS/EOS control IDs.

vllm — Performance, Quantization & Capacity Sizing Guide

vllm — Dense & Sparse Mixture-of-Experts Inference Engine

1. Performance Design Architecture

Autoregressive LLM token decoding is fundamentally memory-bandwidth bound: every generated token must stream active layer weights from DRAM or Unified Memory into vector registers. vllm maximizes throughput and minimizes memory footprint across four hardware-sympathetic layers:

  1. Zero-Copy OS Virtual Memory Mapping (mmap / CreateFileMappingW): Multi-gigabyte GGUF v2/v3 files map directly into virtual memory in tens of milliseconds (28 ms for 4.02 GB OLMoE-1B-7B) with zero heap duplication.
  2. Hardware Integer & FMA SIMD Kernels (AVX2/FMA/F16C & ARM64 NEON): Input activations are quantized once per projection into Q8_0 + Q8Meta sub-sums so quantized weights (Q4_0..Q8_0, Q4_K, Q5_K, Q6_K) execute directly via 256-bit AVX2 integer multiply-accumulate (_mm256_maddubs_epi16) and FMA lanes without full-matrix FP32 dequantization.
  3. Sparse Mixture-of-Experts (64 -> Top-8) & Expert-Sorted Batch Prefill: Sparse MoE models activate only 8 of 64 routed experts per token (skipping 87.5% of FFN bytes), while batched prefill groups (token, expert) pairs by expert_id so each active expert streams through CPU L3 cache once across an entire prompt batch.
  4. Dynamic Vulkan 1.3 UMA SPIR-V Compute (Wave64 FP16/FP32): On integrated APUs and Vulkan GPUs, embedded 64-lane subgroup reduction shaders execute matrix-vector projections directly over host-visible coherent Unified Memory (1.85x speedup in FP16 over FP32 at [2048 x 2048]).

2. Measured Hardware Benchmarks

Representative live measurements captured on an AMD Ryzen AI / Radeon(TM) 890M Unified Memory (15.56 GB shared UMA heap, 12 threads, Windows x64) workstation:

2.1 Zero-Copy Startup & Model Inference Throughput

Workload / Model Architecture & Quantization Mapped Footprint Cold Mmap Time Measured Throughput / Latency
Qwen2.5-0.5B-Instruct Dense 24L, 14Q / 2KV (7:1 GQA), Q4_K_M (q5_0 / q4_k / q6_k) 468.64 MB 48 ms 30.06 tok/s prompt prefill &bull; 19.46 tok/s autoregressive decode (CPU AVX2/FMA)
OLMoE-1B-7B-0924-Instruct Sparse MoE 16L, 64 -> Top-8 experts (~1.3B active / 6.9B total), Q4_K_M 4,018.32 MB 28 ms 6.1–6.5 tok/s uncontended decode streaming only 12.5% of routed FFN experts per token
Vulkan 1.3 Wave64 MatVec (FP32) [2048 x 2048] cached SPIR-V shader (20-dispatch mean, Radeon 890M UMA) 16.0 MB buffer < 5 ms 2,531.5 us per dispatch (~3.31 GB/s effective single-matrix latency)
Vulkan 1.3 Wave64 MatVec (FP16) [2048 x 2048] cached SPIR-V shader (20-dispatch mean, Radeon 890M UMA) 8.0 MB buffer < 5 ms 1,367.6 us per dispatch (1.85x faster than FP32, zero PCIe copy)

vllm Vulkan 1.3 Unified Memory & Wave64 FP16/FP32 Benchmark


2.2 Multi-Turn GQA Prefix KV-Cache Reuse

In multi-turn vllm chat and vllm serve (POST /v1/chat/completions) workloads, vllm stores past Key and Value vectors in FP16 and compares incoming prompt token arrays against its cached prefix:

Session Turn (vllm serve on 127.0.0.1:18510) Total Prompt Tokens Cached Prefix Hit (kv_hit) Newly Prefilled Tokens Prefill Compute Saved
Turn 1 (Initial System + User Question) 34 tokens 0 tokens 34 tokens (100%) Baseline initial prefill
Turn 2 (Follow-Up Question + Assistant History) 117 tokens 35 tokens (System + Turn 1 User prefix) 82 tokens (70.1%) 29.9% prefill skipped
Long System Prompt / RAG Context (1,500 tok prefix + 40 tok query) 1,540 tokens 1,500 tokens 40 tokens (2.6%) 97.4% prefill skipped

3. Quantization Formats & Memory Sizing Reference

vllm supports 11 native tensor formats directly over memory-mapped blocks without heap unpacking:

DType Block Size (Weights) Bytes per Block Effective Bits / Weight (bpw) Typical Accuracy / Bandwidth Trade-Off
Q4_0 32 18 B (f16 d + 16 B nibbles) 4.50 bpw Fastest legacy 4-bit layout; minimum memory footprint.
Q4_K (Q4_K_M) 256 144 B (f16 d, dmin + 12 B scales + 128 B qs) 4.50 bpw Recommended production default — 6-bit sub-block scales + min offsets preserve near-FP16 perplexity.
Q4_1 32 20 B (f16 d, m + 16 B nibbles) 5.00 bpw Legacy 4-bit layout with explicit per-block minimum.
Q5_0 32 22 B (f16 d + 4 B qh + 16 B qs) 5.50 bpw High-precision 5-bit attention/FFN blocks (heavily used in compact Qwen2.5-0.5B Q4_K_M mixes).
Q5_K (Q5_K_M) 256 176 B (f16 d, dmin + 12 B scales + 32 B qh + 128 B qs) 5.50 bpw K-Quant 5-bit super-block for sensitive reasoning and coding models.
Q6_K 256 210 B (128 B ql + 64 B qh + 16 B scales + f16 d) 6.56 bpw Near-lossless quantization used for output.weight LM heads and sensitive down-projections.
Q8_0 32 34 B (f16 d + 32 x int8) 8.50 bpw Virtually indistinguishable from FP16; also used on-the-fly for quantized activation scratchpads.
F16 / BF16 1 (dense) 2 B 16.00 bpw Unquantized half-precision (IEEE-754 binary16 and bfloat16); requires ~3.5x more RAM than Q4_K.
F32 1 (dense) 4 B 32.00 bpw Used selectively for tiny 1D RMSNorm, QK-Norm, attention biases, and MoE router gates.

4. End-to-End RAM & KV-Cache Capacity Formulas

Total steady-state memory required by a vllm instance consists of three deterministic components:

  1. Mapped Model Weights (M_weights):

$$\text{M\_weights} \approx \frac{\text{Total Parameters} \times \text{bpw}}{8}$$

  1. FP16 Grouped-Query Attention KV Cache (M_kv):

$$\text{M\_kv (bytes)} = 2 \times \text{n\_layers} \times \text{ctx\_len} \times \text{n\_kv\_heads} \times \text{head\_dim} \times 2\text{ bytes (FP16)}$$

  1. Activation & Quantized Scratch Arena (M_scratch):
    • Typically 32 MB – 160 MB (pre-allocated once for single-token state, batch prefill buffers, RoPE cos/sin tables, and Q8_0 scratch).

Concrete Capacity Planning Table (--ctx 4096)

Model Class Params (Active / Total) Q4_K_M Weights Q8_0 Weights BF16 Weights FP16 KV Cache (@ 4096 ctx) Recommended Host RAM (Q4_K_M)
Qwen2.5-0.5B (24L, 2 KV, d_h=64) 0.5B / 0.5B ~0.46 GB ~0.53 GB ~0.99 GB 24.0 MB (7:1 GQA + FP16) 2 GB RAM
Qwen2.5-3B (36L, 2 KV, d_h=128) 3.1B / 3.1B ~1.93 GB ~3.29 GB ~6.18 GB 72.0 MB (8:1 GQA + FP16) 4 GB RAM
OLMoE-1B-7B (16L, 16 KV, d_h=128) 1.3B / 6.9B ~4.02 GB ~7.33 GB ~13.80 GB 512.0 MB (16 KV heads, FP16) 8 GB RAM (16 GB for UMA GPU)
LLaMA-3.1-8B (32L, 8 KV, d_h=128) 8.0B / 8.0B ~4.68 GB ~8.50 GB ~16.05 GB 512.0 MB (4:1 GQA + FP16) 8–16 GB RAM

5. Production Tuning Checklist

  1. Match --threads to Physical Cores: On an 8-core / 16-thread or 12-core processor, test --threads 8 vs --threads 12 using ./vllm bench. Excess hyperthreads competing for the same L2/L3 cache and DDR5 memory channels can slightly reduce decode tok/s.
  2. Right-Size --ctx to Actual Workload: Allocating --ctx 32768 when your application sends <= 2048-token prompts reserves 8x more FP16 KV cache than needed; set --ctx 2048 or --ctx 4096 for chat microservices to keep the KV arena inside fast CPU caches.
  3. Prefer Q4_K_M or Q5_K_M for Decode Speed: Because token generation is DRAM-bandwidth bound, Q4_K_M (4.50 bpw) typically decodes 2.8x – 3.4x faster than BF16/F16 (16.0 bpw) on CPUs and integrated APUs while retaining > 99% downstream task quality.

vllm — Consolidated End-User, Administrator & Operations Handbook

vllm — Dense & Sparse Mixture-of-Experts Inference Engine


Part I: vllm — Dense & Sparse Mixture-of-Experts (MoE) Inference Engine

Overview

vllm is a zero-dependency, single-binary Dense and Sparse Mixture-of-Experts (MoE) Large Language Model Inference Engine and OpenAI-Compatible Serving Runtime. Distributed as a self-contained native executable for Windows, Linux, and macOS, vllm maps multi-gigabyte GGUF v2/v3 model files directly into virtual memory in milliseconds and executes quantized inference across multi-threaded x86_64 AVX2/FMA/F16C, ARM64 NEON, and Vulkan 1.3 Unified Memory Architecture (UMA) compute shaders.

SERVING & INFERENCE TOPOLOGY

End-to-End Inference & OpenAI-Compatible HTTP / SSE Serving Topology

Click any component node to inspect how client requests traverse tokenization, KV-cached transformer execution, sampling, and UTF-8 streaming.

Interactive SVG • Click any node (7 components)
1. CLIENT & CLI INGRESS SDKs / curl / Interactive CLI 2. PROTOCOL, TOKENIZATION & INFERENCE CORE HTTP/1.1 + ChatML BPE + Stateful Transformer 3. SAMPLING & STREAM DECODE Top-K/Top-P/Min-P + UTF-8 SSE HTTP / JSON Prompt / REPL Messages Token IDs Vocab Logits Next Token API Clients OpenAI SDKs & Agents POST /v1/chat/completions Operator CLI CLI Subcommands run, chat, serve, bench, gpu Protocol Core OpenAI HTTP/SSE Server Zero-Reflection JSON & SSE Tokenization ChatML & BPE Tokenizer GPT-2 Byte-Level + SentencePiece Compute Core Transformer Runtime Zero-Alloc Forward & Batch Prefill Sampling Logit Sampler Chain Repetition -> Top-K -> Top-P -> Min-P Output Stream UTF-8 Stream Decoder Multi-Byte Codepoint Buffer
Components:
API Clients
OpenAI SDKs & Agents
External OpenAI Client Boundary
Component 1 of 7
Architectural Role & Purpose

External applications, IDE assistants, Python/JS OpenAI SDKs, and curl clients consuming local LLM endpoints.

Contextual Description & System Interaction

Connects over loopback (127.0.0.1 by default) or an authenticated API gateway without external cloud dependencies.

Core Functionality & Execution Mechanics
  • Submits synchronous JSON or Server-Sent Events (stream=true) chat requests
  • Queries GET /v1/models and GET /health for readiness and model discovery
  • Sends plain string messages or multimodal OpenAI content-part arrays
Operator CLI
CLI Subcommands
Command-Line Control Surface
Component 2 of 7
Architectural Role & Purpose

Direct terminal workflow entrypoints for single-turn generation, multi-turn REPL chat, GGUF inspection, and GPU probing.

Contextual Description & System Interaction

Shares the exact same zero-allocation inference and tokenizer pipeline used by the HTTP serving daemon.

Core Functionality & Execution Mechanics
  • Executes interactive multi-turn sessions with /reset and /exit commands
  • Runs deterministic prefill/decode throughput benchmarks via bench
  • Automatically fetches missing GGUF artifacts over HTTPS TLS 1.2+ staging
Protocol Core
OpenAI HTTP/SSE Server
OpenAI HTTP & SSE Gateway
Component 3 of 7
Architectural Role & Purpose

Parses bounded HTTP/1.1 requests with a recursive-descent JSON parser and streams incremental SSE chunks.

Contextual Description & System Interaction

Eliminates dynamic reflection overhead while maintaining wire compatibility with OpenAI chat completion schemas.

Core Functionality & Execution Mechanics
  • Enforces 30s read / 60s write socket timeouts, 64 KiB header cap, and 1 MiB body limit
  • Limits JSON nesting depth to 32 levels to prevent stack exhaustion
  • Flushes data: {...} SSE delta frames on every decoded token boundary
Tokenization
ChatML & BPE Tokenizer
ChatML & BPE Tokenizer Subsystem
Component 4 of 7
Architectural Role & Purpose

Formats system/user/assistant turns with injection-safe ChatML markers and encodes text into integer token IDs.

Contextual Description & System Interaction

Guarantees structural separation between trusted conversation framing and untrusted user prompt content.

Core Functionality & Execution Mechanics
  • Encodes untrusted user text via ordinary BPE so control tokens cannot be spoofed
  • Supports GPT-2 256-byte unicode tables (Qwen2/OLMoE) and SentencePiece styles
  • Applies priority-ranked BPE merge pairs directly from GGUF vocabulary arrays
Compute Core
Transformer Runtime
Stateful Transformer Inference Runtime
Component 5 of 7
Architectural Role & Purpose

Executes dense (LLaMA/Qwen2/Mistral/Gemma) and sparse MoE (OLMoE) transformer layers over pre-allocated arenas.

Contextual Description & System Interaction

Coordinates quantized weight views, GQA attention, sparse expert routing, CPU SIMD workers, and Vulkan UMA.

Core Functionality & Execution Mechanics
  • Matches longest shared token prefix in KV cache to skip redundant prefill work
  • Dispatches batched prefill across prompt deltas and single-token autoregressive decode
  • Executes with zero heap allocations per generated token across all layers
Sampling
Logit Sampler Chain
Non-Mutating Logit Sampler Pipeline
Component 6 of 7
Architectural Role & Purpose

Selects the next token ID deterministically (greedy) or stochastically from vocabulary logits without mutating model state.

Contextual Description & System Interaction

Prevents subtle logit-aliasing bugs across multi-turn steps while supporting creative and deterministic decoding.

Core Functionality & Execution Mechanics
  • Copies logits into a private scratch buffer before applying repetition penalties
  • Chains temperature scaling, partial-sort Top-K, nucleus Top-P, and Min-P filtering
  • Drives reproducible sampling via seeded xoshiro PRNG state
Output Stream
UTF-8 Stream Decoder
Incremental UTF-8 Stream Decoder
Component 7 of 7
Architectural Role & Purpose

Reassembles byte-level BPE pieces across token boundaries so partial UTF-8 sequences never corrupt client output.

Contextual Description & System Interaction

Ensures CJK characters, mathematical symbols, and multi-byte sequences stream cleanly over CLI stdout and HTTP SSE.

Core Functionality & Execution Mechanics
  • Buffers incomplete 2-byte, 3-byte, and 4-byte UTF-8 lead/continuation bytes
  • Emits clean UTF-8 strings immediately once codepoints complete
  • Validates EOS and ChatML stop tokens (<|im_end|>, <|endoftext|>) per step

Core Capabilities

Audience Capability Operational Value
Application Developers Drop-In OpenAI REST & SSE Server (serve) Point official OpenAI Python/Node.js SDKs, IDE coding assistants, local RAG pipelines, and curl clients at http://127.0.0.1:8080/v1/chat/completions for synchronous JSON or real-time text/event-stream SSE token streaming.
AI & ML Practitioners Dense + Sparse MoE GGUF Execution (run, chat) Run compact dense GQA models (Qwen2.5, LLaMA 3, Mistral, Gemma, Phi) and sparse 64 -> Top-8 Mixture-of-Experts models (OLMoE-1B-7B) directly from quantized GGUF files (Q4_K_M, Q5_0, Q6_K, Q8_0, BF16, F16, F32) with zero Python or CUDA runtime dependencies.
Multi-Turn Chat Users Automatic Prefix KV-Cache Reuse Consecutive chat turns automatically reuse the longest matching token prefix stored in the compact FP16 KV Cache, prefilling only newly appended user messages for snappy follow-up responses.
Platform & Edge Operators Zero-Copy Virtual Memory + Vulkan 1.3 UMA Cold-start a 4.0 GB MoE model in ~28 ms via read-only OS memory mapping (CreateFileMappingW / mmap) and accelerate large matrix projections on integrated APUs or discrete GPUs without installing a Vulkan SDK.
Security Teams Loopback Isolation & Hardened Binary Guards Default 127.0.0.1 binding, strict GGUF offset/overflow validation, injection-safe ChatML ordinary-token encoding, and bounded HTTP/JSON framing protect air-gapped workstations and enterprise inference nodes.

Live Diagnostics & Serving Previews

Multi-Turn OpenAI Chat & Streaming SSE Session

vllm Live Multi-Turn OpenAI Chat Session with Prefix KV-Cache Reuse

Zero-Copy GGUF v3 Model & Quantized Tensor Inspector

vllm Zero-Copy GGUF v3 Model & Tensor Catalog Inspector

Dynamic Vulkan 1.3 Unified Memory & Wave64 Shader Probe

vllm Dynamic Vulkan 1.3 Unified Memory & SPIR-V Shader Probe


End-User Documentation Suite

  1. Overview & Readme — Product overview, supported model families, live previews, and quickstart commands.
  2. How-To Guide — Step-by-step recipes for inspecting GGUF models, running interactive chat, serving OpenAI SSE endpoints, enabling Vulkan GPU compute, and connecting SDKs.
  3. End-User Guide — Prompting workflows, multi-turn ChatML sessions, sampling controls (--temp, --top-k, --top-p, --min-p, --repeat-penalty), and choosing between Dense and Sparse MoE models.
  4. Administrator Guide — Production installation, memory & KV-cache sizing, reverse-proxy TLS and Bearer token authentication, service management, and air-gapped model staging.
  5. Operations, Use Cases & Examples — Real-world deployment topologies and runnable curl, Python OpenAI SDK, and JavaScript/TypeScript streaming client examples.
  6. Configuration & CLI Reference — Complete reference for all CLI subcommands, runtime flags, GGUF metadata keys, and HTTP/SSE wire schemas.
  7. Performance & Capacity Guide — Measured SIMD (AVX2/FMA/NEON) and Vulkan Wave64 UMA throughput, Q4_K/Q8_0/BF16 memory sizing formulas, and GQA KV-cache prefix reuse benchmarks.
  8. Consolidated User Handbook — Single-volume master compendium combining all End-User, How-To, Administrator, Operations, Configuration, and Performance guides.

Quickstart

Launch vllm using the prebuilt binary for your platform (dist/bin/windows/vllm.exe, dist/bin/linux/vllm, or dist/bin/macos/vllm):

BASH
# 1. Probe Vulkan 1.3 GPU / APU unified memory heaps and Wave64 SPIR-V shader latency
./vllm gpu

# 2. Inspect GGUF v2/v3 architecture hyperparameters and quantized tensor distribution
./vllm inspect models/olmoe-1b-7b-0924-instruct-q4_k_m.gguf

# 3. Generate a single-turn instruction response using ChatML formatting
./vllm run models/qwen2.5-0.5b-instruct-q4_k_m.gguf --chat -p "Explain zero-copy memory mapping in two sentences." -n 96

# 4. Launch an interactive multi-turn terminal chat session
./vllm chat models/olmoe-1b-7b-0924-instruct-q4_k_m.gguf --system "You are a concise systems architect."

# 5. Start the OpenAI-compatible HTTP & SSE server on loopback port 8080
./vllm serve models/qwen2.5-0.5b-instruct-q4_k_m.gguf --host 127.0.0.1 --port 8080 --ctx 4096

Part II: vllm — Step-by-Step How-To Guide

Overview

This guide provides task-oriented recipes for End Users, Application Developers, and System Administrators operating vllm across local workstations, integrated APU systems, and private inference servers.


Recipe 1: Inspect a GGUF Model Before Deployment

Before loading an unfamiliar GGUF file into production memory, inspect its architecture metadata, context ceiling, attention head layout, expert routing configuration, and quantization distribution.

  1. Run the inspect subcommand against any .gguf v2 or v3 artifact:
BASH
./vllm inspect models/olmoe-1b-7b-0924-instruct-q4_k_m.gguf
  1. Review the structured summary emitted in milliseconds via zero-copy OS memory mapping:
TEXT
Mapped 4018.32 MB in 28 ms (GGUF v3, 195 tensors, 35 metadata KV pairs)
=== Model Summary ===
  Architecture : olmoe (AllenAI OLMoE 1B 7B 0924 Instruct)
  Layers       : 16
  Hidden Dim   : 2048
  Attn Heads   : 16 Q / 16 KV (head_dim=128, rope_theta=10000.0)
  Context Len  : 4096
  Vocab Size   : 50304
  MoE Experts  : 8 active / 64 total (moe_inter_dim=1024, shared=0)
  Data Offset  : 0x5c3720 (alignment=32 bytes)
  Tensor Types : {'q4_k': 97, 'f32': 81, 'q6_k': 17}
  1. Verify that Context Len, Attn Heads, and Tensor Types match your host RAM budget (see the Performance & Capacity Guide for exact sizing formulas).

Recipe 2: Run Deterministic vs Creative Single-Turn Prompts

Use vllm run for scripted CLI pipelines, automated evaluations, and shell one-liners.

  1. Deterministic Greedy Decoding (--temp 0) — ideal for code generation, structured extraction, and regression testing:
BASH
./vllm run models/qwen2.5-0.5b-instruct-q4_k_m.gguf \
  --chat \
  --system "Reply with valid JSON only." \
  -p "List three advantages of FP16 KV caching as a JSON array of strings." \
  --temp 0 \
  -n 128
  1. Reproducible Stochastic Sampling (--seed + --top-k + --top-p + --min-p) — ideal for natural prose while maintaining exact reproducibility across runs:
BASH
./vllm run models/olmoe-1b-7b-0924-instruct-q4_k_m.gguf \
  --chat \
  -p "Summarize why Sparse Mixture-of-Experts models achieve 7B quality at 1B compute cost." \
  --temp 0.6 \
  --top-k 40 \
  --top-p 0.90 \
  --min-p 0.05 \
  --repeat-penalty 1.10 \
  --seed 42 \
  -n 160

Recipe 3: Conduct Multi-Turn Interactive REPL Sessions

Use vllm chat to hold stateful multi-turn conversations in your terminal with incremental KV-cache reuse across turns.

  1. Start an interactive session with a custom system prompt and context window:
BASH
./vllm chat models/qwen2.5-0.5b-instruct-q4_k_m.gguf \
  --ctx 4096 \
  --temp 0.4 \
  --system "You are a concise Linux and network engineering assistant."
  1. Enter questions at the >>> prompt. Notice that follow-up turns prefill only the newly added user message because prior conversation tokens remain resident in the FP16 KV cache.
  2. Use built-in session commands at any time:
    • Type /reset to clear conversation history and reset KV cache position to 0.
    • Type /exit or /quit (or press Ctrl+D) to close the session cleanly.

Recipe 4: Probe & Enable Vulkan 1.3 Unified Memory GPU Acceleration

On laptops, mini-PCs with integrated APUs (such as AMD Radeon 890M / Ryzen AI), or discrete Vulkan GPUs, verify driver readiness before enabling --device gpu.

  1. Run the built-in Vulkan compute probe:
BASH
./vllm gpu
  1. Confirm that vllm detects your Vulkan 1.2/1.3 device, lists available memory heaps, and passes the [2048 x 2048] Wave64 FP32 and FP16 matrix-vector verification benchmark (y[0]=767.06).
  2. Enable GPU acceleration on run, chat, bench, or serve with --device gpu:
BASH
./vllm run models/qwen2.5-0.5b-instruct-q4_k_m.gguf \
  --device gpu \
  --chat \
  -p "Explain unified memory architecture on modern APUs." \
  -n 128

Recipe 5: Serve OpenAI-Compatible Endpoints for Local SDKs & IDE Tools

Expose any local GGUF model over the standard OpenAI /v1/chat/completions HTTP and Server-Sent Events (SSE) protocol.

  1. Start the server bound to loopback (127.0.0.1) on port 18510:
BASH
./vllm serve models/qwen2.5-0.5b-instruct-q4_k_m.gguf \
  --host 127.0.0.1 \
  --port 18510 \
  --ctx 4096 \
  --threads 12
  1. Verify server health and model catalog readiness:
BASH
curl -s http://127.0.0.1:18510/health
curl -s http://127.0.0.1:18510/v1/models
  1. Stream real-time SSE token deltas with curl -N:
BASH
curl -N http://127.0.0.1:18510/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-0.5b-instruct-q4_k_m.gguf",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "Give three tips for low-latency LLM serving."}
    ],
    "temperature": 0.3,
    "max_tokens": 120,
    "stream": true
  }'

vllm Live OpenAI Wire Protocol & SSE Frame Inspector


Recipe 6: Benchmark Prefill and Decode Throughput

Measure repeatable prompt-prefill (tok/s) and token-decode (tok/s) performance on your hardware before sizing production capacity.

BASH
./vllm bench models/qwen2.5-0.5b-instruct-q4_k_m.gguf -n 128 --threads 12
./vllm bench models/olmoe-1b-7b-0924-instruct-q4_k_m.gguf -n 64 --threads 12

Compare CPU thread counts (--threads 4, --threads 8, --threads 12) against your CPU's physical core count; memory-bandwidth-bound quantized inference typically peaks at the number of physical performance cores rather than hyperthreaded logical threads.


Part III: vllm — End-User Guide

1. Welcome to Local Private Inference

vllm lets you run modern instruction-tuned and base Large Language Models completely offline on your desktop, laptop, or private workstation. Prompts, documents, source code, and generated responses never leave your machine.

Whether you interact through the terminal (vllm run and vllm chat) or connect your favorite OpenAI-compatible desktop chat client, IDE extension, or local automation script to vllm serve, this guide explains how to choose models, shape prompts, and tune generation behavior.


2. Choosing Between Dense and Sparse MoE Models

vllm supports both Dense Transformer families and Sparse Mixture-of-Experts (MoE) architectures out of the box:

Model Family Example GGUF Artifact Disk / RAM Size Best Suited For
Compact Dense GQA (Qwen2.5-0.5B .. 3B) qwen2.5-0.5b-instruct-q4_k_m.gguf ~469 MB (< 800 MB RSS) Instantaneous coding completions, classification, JSON formatting, edge devices, and rapid multi-turn drafting at 20–30+ tok/s on CPU.
Mid-Size Dense GQA (LLaMA 3 8B, Qwen2.5-7B, Mistral 7B) qwen2.5-7b-instruct-q4_k_m.gguf ~4.4 GB (~5.2 GB RSS) Deep reasoning, long-context summarization, multilingual translation, and complex multi-step analysis on 8–16 GB systems.
Sparse Mixture-of-Experts (OLMoE-1B-7B) olmoe-1b-7b-0924-instruct-q4_k_m.gguf ~4.0 GB (~4.5 GB RSS) 7B-class knowledge capacity at ~1.3B active compute per token — routes each token through only 8 of 64 experts, cutting CPU/RAM bandwidth by 87.5% relative to a dense 7B FFN.

vllm Sparse Mixture-of-Experts 64-to-Top-8 Routing Telemetry


3. Working with Interactive Chat (vllm chat)

Interactive mode wraps every exchange in clean multi-turn conversation framing and keeps your conversation context hot in memory:

BASH
./vllm chat models/olmoe-1b-7b-0924-instruct-q4_k_m.gguf \
  --system "Explain concepts clearly using concrete examples." \
  --temp 0.6 \
  --top-p 0.90

Session Lifecycle & Prefix Caching

  1. First Turn: vllm encodes the system instruction and your first user message, prefills those tokens in a parallel batch, and streams the assistant's reply word by word.
  2. Follow-Up Turns: When you ask a follow-up question, vllm detects that the earlier turns are already stored in its FP16 KV Cache and computes only the new user message before streaming the next reply.
  3. Starting Fresh (/reset): When switching topics, type /reset at the >>> prompt to clear the conversation buffer and reclaim the full --ctx window without restarting or re-mapping the model file.
  4. Exiting (/exit or /quit): Ends the session cleanly and releases memory mappings.

4. Understanding Sampling Controls

Tuning sampling flags changes how vllm chooses each token from the model's probability distribution:

Parameter CLI Flag / JSON Field Recommended Range Effect on Output
Temperature --temp / temperature 0.0 – 1.2 (default 0.7) 0.0 selects the single most probable token at every step (deterministic greedy). 0.2–0.4 favors factual, focused answers; 0.7–0.9 encourages varied phrasing and brainstorming.
Top-K --top-k / top_k 1 – 100 (default 40) Restricts sampling to the K highest-probability candidate tokens, preventing low-probability tail noise. Set 0 to disable.
Top-P (Nucleus) --top-p / top_p 0.50 – 1.00 (default 0.90) Dynamically keeps the smallest set of top tokens whose cumulative probability reaches P, adapting candidate breadth to model confidence.
Min-P --min-p / min_p 0.00 – 0.20 (default 0.05 CLI) Filters out any candidate token whose probability is less than min_p * max_prob, maintaining coherence even at higher temperatures.
Repetition Penalty --repeat-penalty / repetition_penalty 1.00 – 1.20 (default 1.10 CLI) Gently down-weights tokens that have already appeared in the conversation (1.0 disables the penalty).
Random Seed --seed / seed Any 64-bit integer (0 = time seed) Setting a fixed non-zero seed (for example --seed 42) produces bit-for-bit repeatable generations for identical prompts and flags.

Quick Preset Profiles

  • Code Generation, SQL & JSON Extraction: --temp 0 --repeat-penalty 1.0
  • Technical Q&A & Summarization: --temp 0.3 --top-k 40 --top-p 0.90 --min-p 0.05
  • Creative Writing & Brainstorming: --temp 0.8 --top-k 60 --top-p 0.95 --min-p 0.05 --repeat-penalty 1.12

5. Connecting Desktop GUIs & Coding Assistants

When vllm serve is running locally, any tool that supports a custom OpenAI Base URL can use your local model:

  1. Start vllm serve:
BASH
./vllm serve models/qwen2.5-0.5b-instruct-q4_k_m.gguf --host 127.0.0.1 --port 8080 --ctx 4096
  1. Configure your client application with:
    • API Base URL: http://127.0.0.1:8080/v1
    • API Key: local-vllm (any non-empty placeholder string accepted by your client when connecting directly over loopback)
    • Model Name: qwen2.5-0.5b-instruct-q4_k_m.gguf (or query http://127.0.0.1:8080/v1/models to auto-populate)

Both standard single-string messages ("content": "Hello") and modern OpenAI array-of-parts messages ("content": [{"type": "text", "text": "Hello"}]) sent by newer agent frameworks and IDE plugins are supported transparently.


Part IV: vllm — System Administrator & Deployment Guide

1. Deployment Architecture Overview

vllm is engineered for zero-runtime-dependency operations:

  • Single Static/Native Binary: No Python virtual environments, PyTorch wheels, CUDA toolkit installations, or C++ shared-library version conflicts.
  • Read-Only Memory-Mapped Weights: Model .gguf files are opened strictly read-only (PAGE_READONLY on Windows, PROT_READ + MAP_PRIVATE on Linux/macOS). Multiple processes can map the same physical RAM pages safely, and power loss can never corrupt model weights.
  • Deterministic Arena Memory: All activation buffers, quantized scratchpads, and FP16 KV caches are pre-allocated at startup for --ctx <N>. During steady-state token generation, vllm performs zero per-token heap allocations, eliminating out-of-memory spikes mid-request.
  • Loopback-First Network Security: vllm serve binds 127.0.0.1 by default so workstation deployments never inadvertently expose an unauthenticated port on Wi-Fi or corporate LAN interfaces.

2. Platform Installation

Windows x64

  1. Extract vllm-windows-x64.zip into an installation directory such as ./vllm/ (your extracted product root directory).
  2. Verify archive integrity against SHA256SUMS.txt:
POWERSHELL
Get-FileHash .\vllm-windows-x64.zip -Algorithm SHA256
  1. Optionally verify Vulkan 1.3 integrated/discrete GPU availability:
POWERSHELL
.\bin\vllm.exe gpu

Linux x86_64

  1. Verify checksum and extract vllm-linux-x64.tar.gz:
BASH
sha256sum -c SHA256SUMS.txt --ignore-missing
mkdir -p ./bin ./models
tar -xzf vllm-linux-x64.tar.gz --strip-components=1
chmod 0755 ./bin/vllm
  1. Create a dedicated unprivileged service account:
BASH
sudo useradd --system --home-dir . --shell nologin vllm
sudo chown -R root:vllm ./models
sudo chmod 0750 ./models
sudo chmod 0440 ./models/*.gguf

macOS (Apple Silicon ARM64 & Intel x86_64)

  1. Extract vllm-macos-src-bundle.tar.gz (cross-compiled standalone C bundle) or vllm-macos-native.tar.gz (when built natively on macOS):
BASH
tar -xzf vllm-macos-src-bundle.tar.gz
cd vllm-macos-src-bundle/bin
  1. If compiling from the standalone macOS C bundle on a target Mac with Xcode Command Line Tools (clang), execute the included helper script:
BASH
chmod +x build-macos.sh
./build-macos.sh
./vllm gpu

3. Air-Gapped & Verified Model Staging

In regulated or air-gapped environments, administrators should pre-stage verified GGUF artifacts rather than relying on on-demand internet downloads:

  1. Download the approved .gguf artifact on a staging bastion over HTTPS (TLS 1.2+).
  2. Record and verify its SHA-256 digest against your internal model registry manifest.
  3. Run ./vllm inspect <path-to-model.gguf> on the staging host to verify header integrity, tensor bounds, and expected quantization layout before copying the file into ./models/.
  4. Mount ./models read-only (ro,nosuid,nodev,noexec) inside containers or systemd units.

4. Hardened Linux systemd Service Configuration

Run vllm serve on loopback under strict systemd sandboxing:

INI
# ./deploy/vllm.service
[Unit]
Description=vllm Zero-Dependency GGUF Inference Server
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=vllm
Group=vllm
WorkingDirectory=.
ExecStart=./bin/vllm serve ./models/olmoe-1b-7b-0924-instruct-q4_k_m.gguf \
  --host 127.0.0.1 \
  --port 18510 \
  --ctx 4096 \
  --threads 8
Restart=on-failure
RestartSec=3

# Hardening Directives
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadOnlyPaths=./models
CapabilityBoundingSet=
LockPersonality=true
RestrictSUIDSGID=true

[Install]
WantedBy=multi-user.target

Enable and verify the unit:

BASH
sudo systemctl daemon-reload
sudo systemctl enable --now vllm.service
curl -fsS http://127.0.0.1:18510/health

5. Reverse-Proxy TLS Termination & API Bearer Authentication

vllm serve intentionally keeps its HTTP parser minimal and binds 127.0.0.1 by default. When exposing vllm across a network segment or team VLAN, place an identity-aware gateway (vgate) or TLS reverse proxy (Caddy / NGINX) in front of 127.0.0.1:18510 to enforce TLS 1.3, Bearer API Key / OIDC JWT authentication, and SSE flush-through (proxy_buffering off).

Example NGINX TLS + Bearer Token Reverse Proxy

NGINX
map $http_authorization $vllm_auth_ok {
    default 0;
    "Bearer YOUR_ROTATED_HIGH_ENTROPY_API_TOKEN" 1;
}

server {
    listen 443 ssl http2;
    server_name llm-internal.corp.example;

    ssl_certificate     ./certs/llm-internal.pem;
    ssl_certificate_key ./certs/llm-internal.key;
    ssl_protocols       TLSv1.2 TLSv1.3;

    client_max_body_size 1m;

    location / {
        if ($vllm_auth_ok = 0) {
            return 401 '{"error":{"message":"Unauthorized API token","type":"auth_error"}}';
        }
        proxy_pass http://127.0.0.1:18510;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_buffering off;
        proxy_cache off;
        proxy_read_timeout 120s;
    }
}

vllm Runtime Security Guardrails & Bounds Enforcement


6. Health Monitoring, Capacity Controls & Troubleshooting

Symptom / Check Diagnostic Command / Cause Resolution
Readiness Probe curl -fsS http://127.0.0.1:18510/health Expect HTTP 200 with {"status":"ok","model":"...","ready":true} (lock-free; never blocked by active inference). Wire this into load balancer or systemd watchdogs.
Prometheus Metrics curl -fsS http://127.0.0.1:18510/metrics Exposes vllm_requests_total, vllm_requests_active, vllm_errors_total, vllm_tokens_generated_total, vllm_kv_prefix_hits_total, and vllm_inference_duration_ms_sum.
HTTP 413 Payload Too Large / HTTP 431 Request body exceeds 1 MiB or headers exceed 64 KiB Trim oversized RAG document chunks before sending to /v1/chat/completions; vllm enforces strict 1 MiB body and 64 KiB header bounds.
Lower-Than-Expected CPU tok/s --threads set equal to logical hyperthreads instead of physical cores Set --threads equal to physical performance cores (e.g. 8 or 12) to prevent L3 cache and DRAM bus contention.
No Vulkan-capable GPU detected Headless VM or missing vendor ICD (vulkan-1.dll / libvulkan.so.1) Install the host GPU driver's Vulkan runtime or omit --device gpu to run on the AVX2/NEON CPU thread pool.
Long Multi-Turn Context Truncation Conversation exceeds --ctx (default 2048 if overridden below model limit) Increase --ctx 4096 or --ctx 8192 at server startup and verify host RAM headroom for the expanded FP16 KV arena.

Part V: vllm — Operations, Use Cases & Client Examples

1. Operational Deployment Patterns

vllm fits four common enterprise and developer deployment patterns without requiring dedicated datacenter GPU clusters:

Topology Target Hardware Model Selection Operational Pattern
1. Developer Loopback Assistant Laptop / Workstation (16–32 GB RAM, AVX2 or Apple Silicon NEON) qwen2.5-0.5b-instruct-q4_k_m.gguf or olmoe-1b-7b-0924-instruct-q4_k_m.gguf vllm serve bound to 127.0.0.1:8080 backing IDE autocomplete, commit message drafting, and local CLI scripts.
2. Integrated APU Unified-Memory Node Mini-PC / Edge Workstation with Vulkan 1.3 UMA APU (16–64 GB shared RAM) Q4_K_M / Q6_K Dense or Sparse MoE models --device gpu uploads resident projections into host-visible coherent UMA memory with zero PCIe transfer overhead.
3. Local RAG & Knowledge Synthesis Sidecar Co-located Linux service alongside vkbm / vstore Instruction-tuned Q4_K_M or Q8_0 GGUF Synthesizes grounded answers over retrieved knowledge chunks strictly over loopback HTTP (127.0.0.1:18510) with zero external data egress.
4. Team Gateway Behind vgate / NGINX 8–16 physical core server (32–64 GB RAM) OLMoE-1B-7B (Q4_K_M) or 7B–8B Dense GQA Reverse proxy terminates TLS 1.3 and validates OIDC / Bearer tokens before forwarding /v1/chat/completions over loopback.

2. Runnable Integration Examples

Start a local vllm serve instance before running the client examples below:

BASH
./vllm serve models/qwen2.5-0.5b-instruct-q4_k_m.gguf --host 127.0.0.1 --port 18510 --ctx 4096

vllm Live Multi-Turn Session & Prefix KV-Cache Telemetry


Example A: curl Synchronous JSON & Streaming SSE

1. Non-Streaming Completion (stream: false)

BASH
curl -s http://127.0.0.1:18510/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-0.5b-instruct-q4_k_m.gguf",
    "messages": [
      {"role": "system", "content": "You are a concise systems engineering assistant."},
      {"role": "user", "content": "Why does zero-copy memory mapping accelerate GGUF startup?"}
    ],
    "max_tokens": 96,
    "temperature": 0.2,
    "seed": 42,
    "stream": false
  }'

2. Streaming Server-Sent Events (stream: true)

BASH
curl -N http://127.0.0.1:18510/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-0.5b-instruct-q4_k_m.gguf",
    "messages": [
      {"role": "system", "content": "You are a concise systems engineering assistant."},
      {"role": "user", "content": "Explain grouped-query attention in three bullet points."}
    ],
    "max_tokens": 128,
    "temperature": 0.3,
    "stream": true
  }'

Example B: Official Python openai SDK (Multi-Turn Prefix Reuse)

Because vllm implements the standard /v1/chat/completions and /v1/models wire contracts, the official openai Python client works with zero code changes beyond setting base_url:

PYTHON
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:18510/v1",
    api_key="local-vllm",
)

messages = [
    {"role": "system", "content": "You are a concise infrastructure reliability engineer."},
    {"role": "user", "content": "What are two benefits of read-only memory-mapped model weights?"},
]

# Turn 1: Synchronous JSON response
turn1 = client.chat.completions.create(
    model="qwen2.5-0.5b-instruct-q4_k_m.gguf",
    messages=messages,
    temperature=0.2,
    max_tokens=96,
)
reply1 = turn1.choices[0].message.content
print("Turn 1:", reply1)

# Turn 2: Append history and stream SSE deltas (automatically hits vllm's prefix KV cache!)
messages.append({"role": "assistant", "content": reply1})
messages.append({"role": "user", "content": "How does FP16 KV caching complement that?"})

stream = client.chat.completions.create(
    model="qwen2.5-0.5b-instruct-q4_k_m.gguf",
    messages=messages,
    temperature=0.2,
    max_tokens=96,
    stream=True,
)

print("Turn 2: ", end="", flush=True)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)
print()

Example C: Node.js / TypeScript Zero-Dependency fetch SSE Reader

Modern Node.js (v20+) and browser runtimes can consume vllm's text/event-stream responses directly using native fetch:

JAVASCRIPT
async function streamChatCompletion() {
  const response = await fetch("http://127.0.0.1:18510/v1/chat/completions", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      model: "qwen2.5-0.5b-instruct-q4_k_m.gguf",
      messages: [
        { role: "system", content: "Answer in plain, direct engineering prose." },
        { role: "user", content: "Summarize Sparse Mixture-of-Experts routing." },
      ],
      temperature: 0.3,
      max_tokens: 100,
      stream: true,
    }),
  });

  if (!response.ok) {
    throw new Error(`HTTP ${response.status}: ${await response.text()}`);
  }

  const decoder = new TextDecoder();
  let buffer = "";
  for await (const chunk of response.body) {
    buffer += decoder.decode(chunk, { stream: true });
    const events = buffer.split("\n\n");
    buffer = events.pop() || "";
    for (const evt of events) {
      const line = evt.trim();
      if (!line.startsWith("data: ") || line === "data: [DONE]") continue;
      const payload = JSON.parse(line.slice(6));
      const tokenText = payload.choices?.[0]?.delta?.content ?? "";
      process.stdout.write(tokenText);
    }
  }
  process.stdout.write("\n");
}

streamChatCompletion();

Example D: Air-Gapped Log Triage & Batch CLI Pipeline

For offline batch classification or security log summarization where no network daemon is needed, invoke vllm run directly inside shell pipelines:

BASH
./vllm run models/olmoe-1b-7b-0924-instruct-q4_k_m.gguf \
  --chat \
  --system "Classify the severity (INFO, WARN, CRITICAL) and summarize root cause in one line." \
  -p "2026-11-14T07:12:04Z kernel: nvme0n1: I/O error, dev nvme0n1, sector 8841920 op 0x0:(READ)" \
  --temp 0 \
  -n 64

3. Operational Runbook Checklist

  1. Pre-Flight Verification:
    • Run ./vllm inspect <model.gguf> to confirm GGUF v2/v3 header validity and expected tensor count.
    • Run ./vllm gpu if deploying with --device gpu to verify Vulkan driver initialization and numerical parity.
  2. Steady-State Operations:
    • Monitor GET /health (HTTP 200 OK), GET /metrics (Prometheus counters), and structured JSON request logs on stdout (ts, req_id, status, prompt_tokens, kv_hit, completion_tokens, duration_ms).
    • High kv_hit counts on multi-turn agent workloads confirm that clients are preserving deterministic system prompts at the start of messages, maximizing prefix KV-cache reuse.
  3. Graceful Upgrade:
    • Stage the new vllm binary alongside the existing release, verify SHA256SUMS.txt, drain active HTTP connections at the reverse proxy, and restart the service; read-only .gguf files require zero migration or conversion across binary upgrades.

Part VI: vllm — Configuration & CLI Reference

1. Command-Line Synopsis

TEXT
vllm <command> [model.gguf] [options]

If [model.gguf] is omitted on inspect, run, chat, serve, or bench, vllm defaults to models/olmoe-1b-7b-0924-instruct-q4_k_m.gguf and automatically downloads it over HTTPS (TLS 1.2+) if not already present on disk.


2. Subcommands

Subcommand Arguments Description
download <preset-id> [--sha256 <hex>] Downloads an official OLMoE or Qwen GGUF preset over HTTPS (TLS 1.2+) into ./models/, validates GGUF header/tensor bounds and optional SHA-256 digest before atomic rename.
inspect <model.gguf> Memory-maps the GGUF v2/v3 file read-only, validates all metadata and tensor bounds, and prints architecture hyperparameters, expert counts, data alignment, and per-DType tensor counts.
gpu (none) Dynamically probes the system's Vulkan 1.2/1.3 driver, prints GPU/APU memory heaps, verifies numerical parity (y[0]=767.06), and benchmarks [2048 x 2048] Wave64 FP32 and FP16 SPIR-V matrix-vector dispatches.
run <model.gguf> [options] Executes a single prompt prefill + autoregressive token generation pass and prints prefill/decode throughput (tok/s) upon completion.
chat <model.gguf> [options] Launches an interactive multi-turn ChatML REPL session with automatic prefix KV-cache reuse and /clear, /exit, /quit session commands.
serve <model.gguf> [options] Starts the concurrent OpenAI-compatible HTTP/1.1 & Server-Sent Events (SSE) server exposing /health, /metrics, /v1/models, and /v1/chat/completions with graceful SIGINT/SIGTERM shutdown.
help [-h | --help | -v | --version] Prints command usage syntax, supported model presets, or version metadata (vllm 0.1.0).

3. Complete Runtime Flags Reference

Flag Alias Value Type Default Applicable Commands Description
--prompt -p string "Hello" run Input prompt text to complete (or initial user message when combined with --chat).
--chat — (boolean) false (run), true (chat) run Wraps --system and --prompt in injection-safe ChatML <|im_start|> / <|im_end|> conversation framing.
--system -s string "" run --chat, chat System instruction placed at the head of the ChatML conversation context.
--max-tokens -n int (>= 1) 128 run, chat, serve Maximum number of completion tokens generated per turn or request.
--ctx -c int (>= 16) 2048 (clamped to <= model.context_length) run, chat, serve Maximum KV-cache sequence length pre-allocated at startup.
--threads — int (1..256) Auto-detected physical core count run, chat, serve Number of persistent worker threads in the lock-free SIMD work-stealing thread pool.
--device — cpu | gpu cpu run, chat, serve Selects multi-threaded CPU SIMD execution (cpu) or Vulkan UMA/VRAM GPU offload (gpu) with automatic CPU fallback.
--gpu-index — int (>= 0) Auto (prefers hardware GPU) run, chat, serve Explicit Vulkan physical device index (also configurable via VLLM_GPU_INDEX).
--temp -t f32 (>= 0.0) 0.7 run, chat, serve Sampling temperature (0.0 switches to deterministic argmax greedy decoding).
--top-k — int (>= 0) 40 run, chat, serve Top-K candidate cutoff before nucleus filtering (0 disables Top-K).
--top-p — f32 ((0.0, 1.0]) 0.90 run, chat, serve Nucleus cumulative probability threshold (1.0 disables Top-P).
--min-p — f32 ([0.0, 1.0]) 0.0 run, chat, serve Minimum candidate probability relative to the top token probability (0.0 disables Min-P).
--repeat-penalty — f32 (> 0.0) 1.0 run, chat, serve Multiplicative penalty applied to tokens already present in the last 64 tokens (1.0 disables).
--seed — u64 0 (wall-clock seed) run, chat, serve PRNG seed for deterministic stochastic sampling when non-zero.
--host — string "127.0.0.1" serve IPv4 bind address for the OpenAI HTTP server. Defaults to loopback 127.0.0.1 for local isolation.
--port — int (1..65535) 8080 serve TCP port for the OpenAI HTTP & SSE server.
--api-key — string "" (or VLLM_API_KEY env) serve Optional constant-time Bearer API key required for /v1/* and /metrics endpoints.
--cors-origin — string "" (disabled) serve Opt-in Access-Control-Allow-Origin header value. Disabled by default to prevent localhost CSRF.
--no-prefix-cache — (boolean) false serve Disables cross-request KV prefix cache reuse for multi-tenant isolation.
--max-connections — int (1..4096) 64 serve Maximum concurrent TCP connections accepted by serve (excess connections receive HTTP 503).
--sha256 — string (64 hex) "" download Verifies cryptographic SHA-256 digest of the downloaded .gguf artifact before atomic rename.

4. OpenAI HTTP & SSE API Reference (vllm serve)

4.1 Endpoints Summary

Method Path Response Content-Type Description
GET /health application/json Lock-free liveness and readiness probe ({"status":"ok","model":"<id>","ready":true} or HTTP 503 during graceful shutdown).
GET /metrics text/plain; version=0.0.4 Lock-free Prometheus metrics (vllm_requests_total, vllm_requests_active, vllm_errors_total, vllm_tokens_generated_total, vllm_kv_prefix_hits_total, vllm_inference_duration_ms_sum).
GET /v1/models application/json OpenAI model list catalog returning the active GGUF model ID and owned_by: "vllm".
POST /v1/chat/completions application/json or text/event-stream OpenAI chat completion endpoint supporting multi-turn messages, stop sequences, Transfer-Encoding: chunked, Expect: 100-continue, prefix KV-cache reuse, and stream: true SSE deltas (including stream_options.include_usage).
OPTIONS * application/json Preflight handler; emits CORS headers only when --cors-origin is explicitly configured.

4.2 POST /v1/chat/completions Request Schema

JSON Field Type Default Description
model string Loaded model ID Optional model identifier string echoed in responses.
messages array<object> (required, non-empty) Ordered array of {"role": "system"|"user"|"assistant", "content": "..." | [{"type":"text","text":"..."}]} turns. Supports full Unicode including UTF-16 surrogate pairs.
stream boolean false When true, streams incremental data: {...}\n\n SSE chunks terminated by data: [DONE]\n\n.
stream_options object {} Supports {"include_usage": true} to emit a final usage SSE chunk before data: [DONE].
stop string | array<string> [] Up to 16 stop sequences that halt generation cleanly with finish_reason: "stop".
max_tokens integer Server --max-tokens (max_completion_tokens alias supported) Maximum generated tokens (>= 1, automatically clamped to remaining --ctx headroom).
temperature number 0.7 Sampling temperature in [0.0, 100.0] (0.0 = greedy argmax).
top_p number 0.90 Nucleus sampling threshold in (0.0, 1.0].
top_k integer 40 Top-K candidate cutoff (>= 0).
min_p number 0.00 Relative Min-P probability cutoff in [0.0, 1.0].
repeat_penalty number 1.00 Token repetition penalty multiplier in (0.0, 100.0].
seed integer 0 Deterministic PRNG seed (> 0).

4.3 Protocol & Security Limits

Limit Enforced Value Behavior When Exceeded
Total Read / Write Deadlines 30 s total wall-clock read / 60 s write Closes stalled or Slowloris TCP connections cleanly (HTTP 408 Request Timeout).
Max HTTP Request Header 64 KiB (65,536 bytes) Rejects oversized header blocks with HTTP 431 Request Header Fields Too Large.
Max HTTP Request Body 1 MiB (1,048,576 bytes) Rejects oversized Content-Length or chunked streams with HTTP 413 Payload Too Large.
HTTP Smuggling Guard RFC 9112 §6.3 Rejects conflicting duplicate Content-Length or simultaneous Content-Length + Transfer-Encoding with HTTP 400.
Max Concurrent Connections 64 (configurable via --max-connections) Rejects excess connections cleanly with HTTP 503 Service Unavailable.
Max JSON Nesting Depth 32 levels Aborts recursive JSON parsing cleanly with HTTP 400 to prevent stack exhaustion.

5. Self-Describing GGUF Metadata Keys

vllm requires zero external configuration files because all model hyperparameters are resolved directly from standardized GGUF v2/v3 metadata keys at load time:

Metadata Key Pattern (<arch> = llama, qwen2, olmoe, etc.) Resolved Parameter
general.architecture, general.name, general.alignment Model family dispatch, human-readable name, and tensor data byte alignment (32 bytes default).
<arch>.block_count, <arch>.embedding_length, <arch>.context_length Transformer layer count, hidden dimension d_model, and maximum training context window.
<arch>.attention.head_count, <arch>.attention.head_count_kv, <arch>.attention.key_length Query head count, Grouped-Query Attention (GQA) KV head count, and per-head dimension head_dim.
<arch>.rope.freq_base, <arch>.attention.layer_norm_rms_epsilon Rotary Positional Embedding base frequency (theta) and RMSNorm epsilon (1e-5 / 1e-6).
<arch>.expert_count, <arch>.expert_used_count, <arch>.expert_feed_forward_length Sparse MoE total routed experts (64), active Top-K experts per token (8), and expert FFN intermediate width.
tokenizer.ggml.model, tokenizer.ggml.tokens, tokenizer.ggml.merges, tokenizer.ggml.bos_token_id, tokenizer.ggml.eos_token_id Vocabulary style (gpt2 byte-level or llama SentencePiece), token strings, BPE merge priorities, and BOS/EOS control IDs.

Part VII: vllm — Performance, Quantization & Capacity Sizing Guide

1. Performance Design Architecture

Autoregressive LLM token decoding is fundamentally memory-bandwidth bound: every generated token must stream active layer weights from DRAM or Unified Memory into vector registers. vllm maximizes throughput and minimizes memory footprint across four hardware-sympathetic layers:

  1. Zero-Copy OS Virtual Memory Mapping (mmap / CreateFileMappingW): Multi-gigabyte GGUF v2/v3 files map directly into virtual memory in tens of milliseconds (28 ms for 4.02 GB OLMoE-1B-7B) with zero heap duplication.
  2. Hardware Integer & FMA SIMD Kernels (AVX2/FMA/F16C & ARM64 NEON): Input activations are quantized once per projection into Q8_0 + Q8Meta sub-sums so quantized weights (Q4_0..Q8_0, Q4_K, Q5_K, Q6_K) execute directly via 256-bit AVX2 integer multiply-accumulate (_mm256_maddubs_epi16) and FMA lanes without full-matrix FP32 dequantization.
  3. Sparse Mixture-of-Experts (64 -> Top-8) & Expert-Sorted Batch Prefill: Sparse MoE models activate only 8 of 64 routed experts per token (skipping 87.5% of FFN bytes), while batched prefill groups (token, expert) pairs by expert_id so each active expert streams through CPU L3 cache once across an entire prompt batch.
  4. Dynamic Vulkan 1.3 UMA SPIR-V Compute (Wave64 FP16/FP32): On integrated APUs and Vulkan GPUs, embedded 64-lane subgroup reduction shaders execute matrix-vector projections directly over host-visible coherent Unified Memory (1.85x speedup in FP16 over FP32 at [2048 x 2048]).

2. Measured Hardware Benchmarks

Representative live measurements captured on an AMD Ryzen AI / Radeon(TM) 890M Unified Memory (15.56 GB shared UMA heap, 12 threads, Windows x64) workstation:

2.1 Zero-Copy Startup & Model Inference Throughput

Workload / Model Architecture & Quantization Mapped Footprint Cold Mmap Time Measured Throughput / Latency
Qwen2.5-0.5B-Instruct Dense 24L, 14Q / 2KV (7:1 GQA), Q4_K_M (q5_0 / q4_k / q6_k) 468.64 MB 48 ms 30.06 tok/s prompt prefill &bull; 19.46 tok/s autoregressive decode (CPU AVX2/FMA)
OLMoE-1B-7B-0924-Instruct Sparse MoE 16L, 64 -> Top-8 experts (~1.3B active / 6.9B total), Q4_K_M 4,018.32 MB 28 ms 6.1–6.5 tok/s uncontended decode streaming only 12.5% of routed FFN experts per token
Vulkan 1.3 Wave64 MatVec (FP32) [2048 x 2048] cached SPIR-V shader (20-dispatch mean, Radeon 890M UMA) 16.0 MB buffer < 5 ms 2,531.5 us per dispatch (~3.31 GB/s effective single-matrix latency)
Vulkan 1.3 Wave64 MatVec (FP16) [2048 x 2048] cached SPIR-V shader (20-dispatch mean, Radeon 890M UMA) 8.0 MB buffer < 5 ms 1,367.6 us per dispatch (1.85x faster than FP32, zero PCIe copy)

vllm Vulkan 1.3 Unified Memory & Wave64 FP16/FP32 Benchmark


2.2 Multi-Turn GQA Prefix KV-Cache Reuse

In multi-turn vllm chat and vllm serve (POST /v1/chat/completions) workloads, vllm stores past Key and Value vectors in FP16 and compares incoming prompt token arrays against its cached prefix:

Session Turn (vllm serve on 127.0.0.1:18510) Total Prompt Tokens Cached Prefix Hit (kv_hit) Newly Prefilled Tokens Prefill Compute Saved
Turn 1 (Initial System + User Question) 34 tokens 0 tokens 34 tokens (100%) Baseline initial prefill
Turn 2 (Follow-Up Question + Assistant History) 117 tokens 35 tokens (System + Turn 1 User prefix) 82 tokens (70.1%) 29.9% prefill skipped
Long System Prompt / RAG Context (1,500 tok prefix + 40 tok query) 1,540 tokens 1,500 tokens 40 tokens (2.6%) 97.4% prefill skipped

3. Quantization Formats & Memory Sizing Reference

vllm supports 11 native tensor formats directly over memory-mapped blocks without heap unpacking:

DType Block Size (Weights) Bytes per Block Effective Bits / Weight (bpw) Typical Accuracy / Bandwidth Trade-Off
Q4_0 32 18 B (f16 d + 16 B nibbles) 4.50 bpw Fastest legacy 4-bit layout; minimum memory footprint.
Q4_K (Q4_K_M) 256 144 B (f16 d, dmin + 12 B scales + 128 B qs) 4.50 bpw Recommended production default — 6-bit sub-block scales + min offsets preserve near-FP16 perplexity.
Q4_1 32 20 B (f16 d, m + 16 B nibbles) 5.00 bpw Legacy 4-bit layout with explicit per-block minimum.
Q5_0 32 22 B (f16 d + 4 B qh + 16 B qs) 5.50 bpw High-precision 5-bit attention/FFN blocks (heavily used in compact Qwen2.5-0.5B Q4_K_M mixes).
Q5_K (Q5_K_M) 256 176 B (f16 d, dmin + 12 B scales + 32 B qh + 128 B qs) 5.50 bpw K-Quant 5-bit super-block for sensitive reasoning and coding models.
Q6_K 256 210 B (128 B ql + 64 B qh + 16 B scales + f16 d) 6.56 bpw Near-lossless quantization used for output.weight LM heads and sensitive down-projections.
Q8_0 32 34 B (f16 d + 32 x int8) 8.50 bpw Virtually indistinguishable from FP16; also used on-the-fly for quantized activation scratchpads.
F16 / BF16 1 (dense) 2 B 16.00 bpw Unquantized half-precision (IEEE-754 binary16 and bfloat16); requires ~3.5x more RAM than Q4_K.
F32 1 (dense) 4 B 32.00 bpw Used selectively for tiny 1D RMSNorm, QK-Norm, attention biases, and MoE router gates.

4. End-to-End RAM & KV-Cache Capacity Formulas

Total steady-state memory required by a vllm instance consists of three deterministic components:

  1. Mapped Model Weights (M_weights):

$$\text{M\_weights} \approx \frac{\text{Total Parameters} \times \text{bpw}}{8}$$

  1. FP16 Grouped-Query Attention KV Cache (M_kv):

$$\text{M\_kv (bytes)} = 2 \times \text{n\_layers} \times \text{ctx\_len} \times \text{n\_kv\_heads} \times \text{head\_dim} \times 2\text{ bytes (FP16)}$$

  1. Activation & Quantized Scratch Arena (M_scratch):
    • Typically 32 MB – 160 MB (pre-allocated once for single-token state, batch prefill buffers, RoPE cos/sin tables, and Q8_0 scratch).

Concrete Capacity Planning Table (--ctx 4096)

Model Class Params (Active / Total) Q4_K_M Weights Q8_0 Weights BF16 Weights FP16 KV Cache (@ 4096 ctx) Recommended Host RAM (Q4_K_M)
Qwen2.5-0.5B (24L, 2 KV, d_h=64) 0.5B / 0.5B ~0.46 GB ~0.53 GB ~0.99 GB 24.0 MB (7:1 GQA + FP16) 2 GB RAM
Qwen2.5-3B (36L, 2 KV, d_h=128) 3.1B / 3.1B ~1.93 GB ~3.29 GB ~6.18 GB 72.0 MB (8:1 GQA + FP16) 4 GB RAM
OLMoE-1B-7B (16L, 16 KV, d_h=128) 1.3B / 6.9B ~4.02 GB ~7.33 GB ~13.80 GB 512.0 MB (16 KV heads, FP16) 8 GB RAM (16 GB for UMA GPU)
LLaMA-3.1-8B (32L, 8 KV, d_h=128) 8.0B / 8.0B ~4.68 GB ~8.50 GB ~16.05 GB 512.0 MB (4:1 GQA + FP16) 8–16 GB RAM

5. Production Tuning Checklist

  1. Match --threads to Physical Cores: On an 8-core / 16-thread or 12-core processor, test --threads 8 vs --threads 12 using ./vllm bench. Excess hyperthreads competing for the same L2/L3 cache and DDR5 memory channels can slightly reduce decode tok/s.
  2. Right-Size --ctx to Actual Workload: Allocating --ctx 32768 when your application sends <= 2048-token prompts reserves 8x more FP16 KV cache than needed; set --ctx 2048 or --ctx 4096 for chat microservices to keep the KV arena inside fast CPU caches.
  3. Prefer Q4_K_M or Q5_K_M for Decode Speed: Because token generation is DRAM-bandwidth bound, Q4_K_M (4.50 bpw) typically decodes 2.8x – 3.4x faster than BF16/F16 (16.0 bpw) on CPUs and integrated APUs while retaining > 99% downstream task quality.