Pages int8 symmetric quantized vectors (260 B @ 256-D, 132 B @ 128-D) through a bounded 4 KB binary page cache (8.0 MB default RAM budget) backed by a crash-safe write-ahead log (index.wal) and atomic index snapshots (index.idx).
Scales from 18,000 handbook nodes to multi-million-passage corpora within a fixed memory envelope.
2-Turn Coarse-to-Fine WorkingSet Zoom
Loads compact L3/L2 summaries on Turn 1, identifies the highest-relevance sections, and executes zoom_in(L2 -> L1/L0) via $O(1)$ shard byte-offset seeks on Turn 2, rendering sorted TOON-LD @graph tables.
Cuts LLM prompt token consumption by 61.4%–80% compared to flat top-$K$ chunk retrieval.
Deterministic Word-for-Word Citation Verifier
Audits structured citations (@id, doc_id, section_id, printed_page, quoted_span) directly against L0 passage shards using exact and whitespace-normalized substring matching.
Eliminates fabricated section references, wrong page numbers, and paraphrased quotes masquerading as verbatim text.
Clause-by-Clause Contract Gap Auditor
Splits vendor agreements (MSAs, cloud schedules, SLAs) into normative clauses, tags deontic modalities (mandatory, prohibition, supervisory, commitment, permission), and classifies coverage as MATCHED, PARTIAL_REVIEW, or GAP_ALERT.
Evaluates a 40-clause contract in under 90 ms without requiring external GPU infrastructure.
OpenAI Proxy, MCP Server & Role-Separated Web Studio
Serves 22 REST endpoints, an OpenAI-compatible /v1/chat/completions proxy, an 8-tool JSON-RPC 2.0 MCP server (POST /mcp & --mcp-stdio), and a 4-portal browser workspace (/ui/auth, /ui/portal, /ui/delegated, /ui/admin).
Drop-in memory plane for human examiners, vllm inference gateways, vsight SOC analysts, and autonomous MCP agents.
Performance & Sizing — Empirical retrieval ablation, 98.5% printed-page accuracy across 58 booklets, token-efficiency benchmarks, and hardware sizing tables.
How-To Guide — Step-by-Step Operational Recipes
This guide provides copy-pasteable operational recipes for compiling PDF and Markdown corpora into 4-level hierarchical knowledge graphs, building hybrid BM25 + Truncated SVD indices, launching the serve_memory OpenAI and MCP server, auditing contracts for regulatory gaps, and verifying verbatim page citations.
Recipe 1: Ingest PDF Handbooks into a Deterministic Offline Corpus
Use the 5-stage offline compiler when publishing immutable handbook distributions (such as OCC Comptroller's Handbooks, FFIEC IT Examination Booklets, or internal SOP manuals) with a standalone knowledge_base.html explorer and citation-preserving Markdown export.
Prerequisites
Ensure Poppler pdftotext (>= 22.x) is installed on your PATH so build_index can extract layout-preserving page columns without invoking a system shell:
BASH
pdftotext -v
Step 1: Define the Corpus Manifest (corpora/my_policy.json)
Create a JSON configuration file inside corpora/ describing your PDF directory, header/footer rules, and curated starter queries:
Step 2: Sign In via the Role-Separated Authentication Gate
Navigate to http://127.0.0.1:7331/ui/auth to establish a scoped session (Examiner / Compliance Analyst, Corpus Steward, or Global Memory Admin) and inspect live header telemetry (Live Nodes, HNSW, LSA Dims, VVEC Hits).
Step 3: Pin a Tier-3 (L3) Supervisory Axiom
Pinned L3 axioms reserve dedicated capacity inside every WorkingSet so non-negotiable institutional mandates are never evicted during retrieval:
Step 4: Ingest Structured Documents via Strategy-Driven Parsing
Use /ui/delegated (Corpus Steward Portal) or POST /ingest_document to compile handbooks, regulations, or contracts directly into L3 -> L2 -> L1 -> L0 nodes.
BASH
curl -sS -X POST http://127.0.0.1:7331/ingest_document \
-H "Authorization: Bearer ${VKBM_AUTH_TOKEN}" \
-H "Content-Type: application/json" \
-d '{
"doc_id": "occ-2023-17",
"document": "OCC Bulletin 2023-17 Third-Party Relationships",
"strategy": "occ_ffiec_handbook",
"salience": 0.92,
"raw_text": "I. Governance and Risk Management\nPage 8\nA banking organization should adopt risk management practices commensurate with the level of risk and complexity of its third-party relationships.\n\nIII.C.2 Right to Audit and Remediation\nPage 27\nContracts for critical activities must stipulate the banking organization right to audit, review independent SOC 2 Type II reports, and enforce timely remediation."
}'
Step 5: Fit the Corpus-Adaptive Truncated SVD (LSA) Subspace
After ingesting or updating documents, refit the pure-CPU Randomized Truncated SVD model so every L3..L0 node is re-projected into the learned latent subspace:
Configure serve_memory with an upstream OpenAI-compatible inference server (such as local vllm, llama-server, or an enterprise gateway) via environment variables:
Any standard OpenAI SDK or HTTP client can now query http://127.0.0.1:7331/v1/chat/completions. vkbm intercepts the user prompt, executes 2-turn coarse-to-fine associative recall + zoom_in, injects the verified TOON-LD @graph context, calls the upstream model, and runs deterministic citation verification before returning the response:
BASH
curl -sS -X POST http://127.0.0.1:7331/v1/chat/completions \
-H "Authorization: Bearer ${VKBM_AUTH_TOKEN}" \
-H "Content-Type: application/json" \
-d '{
"model": "vkbm-grounded",
"messages": [
{
"role": "user",
"content": "What contractual audit rights are required for critical cloud vendors under OCC 2023-17?"
}
],
"temperature": 0.1
}'
Option B: Model Context Protocol (POST /mcp or --mcp-stdio)
For desktop coding assistants, compliance copilots, or autonomous orchestrators, expose the 8 grounded memory tools over JSON-RPC 2.0:
Recipe 4: Run Automated Contract Gap Analysis & Quote Verification
Step 1: Audit Draft Contract Text Against Regulatory Baselines
Submit draft agreement clauses to POST /audit_contract to classify every normative obligation (MATCHED, PARTIAL_REVIEW, or GAP_ALERT) with exact handbook citations:
BASH
curl -sS -X POST http://127.0.0.1:7331/audit_contract \
-H "Authorization: Bearer ${VKBM_AUTH_TOKEN}" \
-H "Content-Type: application/json" \
-d '{
"contract_name": "Core Ledger Cloud Hosting MSA v3",
"contract_text": "1. Audit Rights. Customer may review Provider annual SOC 2 Type II summary report upon written request, and regulatory agencies may examine critical operations.\n\n2. Subcontractors. Provider may engage fourth-party subcontractors at its sole discretion without prior customer notification.\n\n3. Incident Reporting. Provider shall notify Customer within 24 hours of confirming any unauthorized access to Customer NPI.",
"top_k": 3
}'
Step 2: Verify Every Quoted Span Word-for-Word (POST /verify_citations)
Before publishing an AI-authored compliance memo, pass its structured citations to /verify_citations to confirm that every @id, doc_id, section_id, printed_page, and quoted_span matches the L0 shard byte-for-byte:
BASH
curl -sS -X POST http://127.0.0.1:7331/verify_citations \
-H "Authorization: Bearer ${VKBM_AUTH_TOKEN}" \
-H "Content-Type: application/json" \
-d '{
"citations": [
{
"node_id": "n_8",
"doc_id": "occ-2023-17",
"section_id": "III.C.2",
"printed_page": "27",
"quoted_span": "Contracts for critical activities must stipulate the banking organization right to audit"
}
]
}'
This guide explains how compliance examiners, legal reviewers, security analysts, and technical researchers use vkbm to search multi-handbook corpora, navigate the 4-level hierarchy (Book -> Section -> Rule -> Quote), inspect confidence and printed-page anchors, and audit legal contracts.
Understanding the 4-Level Knowledge Hierarchy
Every document compiled into vkbm is structured as a 4-tier pyramid rather than disconnected flat chunks:
Anchors high-level regulatory scope and keeps non-negotiable rules permanently resident in WorkingSet memory.
L2 (tier = 2)
Section Centroid
Authoritative Table-of-Contents section (e.g., §III.C.2 — Right to Audit and Remediation) with an L2-normalized salience-weighted concept vector.
Provides fast, low-token structural orientation on Turn 1 before drilling down into verbatim clauses.
L1 (tier = 1)
Normative Rule / Obligation
Multi-sentence regulatory mandate, prohibition, or contractual commitment tagged with deontic modality (mandatory, prohibition, supervisory, commitment, permission).
Maps individual contract clauses directly against enforceable regulatory obligations.
L0 (tier = 0)
Verbatim Page-Cited Quote
Exact passage text stored in 64 MB .toon shards with immutable {doc_id, section_id, printed_page, page_index} provenance.
Supplies court- and exam-ready verbatim quotations verified word-for-word against the source page.
Signing In & Role-Scoped Workspaces
Open http://127.0.0.1:7331/ui/auth in any modern desktop browser. vkbm separates daily analytical lookup from corpus mutation and store administration across three workspaces:
Examiner / Compliance Analyst (/ui/portal) — Read-only access to Associative Recall (/recall), 2-Turn Cited Q&A (/ask), Contract Gap Audit (/audit_contract), Citation Verification (/verify_citations), and the CentralIndex Record Browser.
Corpus Steward (/ui/delegated) — Scoped privileges to ingest structured handbooks, regulations, and runbooks (/ingest_document), pin supervisory axioms (/ingest), and load .toon synset packs.
Global Memory Administrator (/ui/admin) — Full store governance including Randomized Truncated SVD refitting (/refit_lsa), Ebbinghaus consolidation (/consolidate), WAL/shard compaction (/compact), runtime vector tuning, and live MCP JSON-RPC inspection.
In the Examiner Portal (/ui/portal), select the Associative Recall & WorkingSet workspace:
Enter a natural-language query, acronym, or exact citation phrase (for example, board oversight critical vendor audit rights subcontractor cyber incident notification).
Set your Token Budget (default 500–650 tokens). vkbm automatically reserves space for pinned L3 axioms first, then loads the highest-scoring L2/L1/L0 nodes via 4-layer HNSW traversal, #syn_* synset expansion, and spreading activation (gamma = 0.25).
Inspect the TOON-LD @graph Context Pane and Record Browser:
Records are grouped deterministically by shape (axiom, booklet, section, obligation, passage) and sorted by @id.
Every L0 passage displays its doc_id, section_id, printed_page (the page number printed in the document footer), and page_index (the 1-based physical PDF page).
When answering complex multi-section regulatory questions, flat RAG systems waste thousands of tokens loading full passages from marginally relevant chapters. vkbm uses 2-Turn Coarse-to-Fine Synthesis:
Turn 1 (Coarse Orientation):vkbm retrieves compact L3 booklet headers and L2 TOC section summaries across the entire store.
Turn 2 (Selective zoom_in):vkbm identifies the highest-scoring L2 section nodes and calls zoom_in, swapping out the coarse summary for its exact L1 obligations and L0 verbatim quotes via $O(1)$ shard byte-offset seeks.
Interactive zoom_in and zoom_out Controls
You can also drive coarse-to-fine navigation manually from the UI or HTTP API:
zoom_in (POST /zoom_in): Expands an L3 booklet into its L2 sections, or expands an L2 section (such as n_3 -> OCC-2023-17 §III.C.2) into its child L1 rules and L0 verbatim quotes.
zoom_out (POST /zoom_out): Collapses child L1/L0 nodes back into their parent L2 summary, freeing WorkingSet token budget for exploring another handbook chapter.
Use the Contract Compliance Audit tab in /ui/portal to screen draft vendor agreements, cloud addenda, or internal policies against your loaded regulatory corpora:
Paste the draft agreement text and assign a descriptive document title.
Click Run Clause Audit. vkbm segments the agreement into discrete clauses, classifies each clause's deontic modality (mandatory, prohibition, supervisory, commitment, permission), and preserves legal negations (not, no, never, without, neither, nor).
Review the per-clause disposition cards:
MATCHED: Strong semantic and lexical alignment with authoritative L1/L0 regulatory rules, accompanied by exact section and printed-page citations.
PARTIAL_REVIEW: Moderate overlap or scope divergence (for example, right-to-audit restricted solely to vendor-authored executive summaries or fourth-party subcontracting lacking prior notice).
GAP_ALERT: No supporting regulatory alignment or direct contradiction with a mandatory obligation—requires legal redlining before execution.
Workflow 4: Offline Single-File Explorer (knowledge_base.html)
When operating on air-gapped examination laptops without running a local HTTP daemon, open the compiled knowledge_base.html file directly from disk (./dist/knowledge_base.html):
100% Self-Contained: All CSS, JavaScript, Truncated SVD projection matrices, and passage records are embedded inside the single HTML file with zero external fonts, scripts, or network calls.
Instant Hybrid Filtering: Type regulatory terms or click curated starter chips to rank passages across booklets, filter by handbook title or TOC section, and copy pre-formatted citations (Document, Section §X.Y, p. N) to your clipboard.
Administrator Guide — Corpus Governance, Security & Operations
This guide covers day-2 operational administration of vkbm: persistent store anatomy, deterministic artifact verification, air-gapped deployment, Bearer authentication and SSRF controls, Truncated SVD subspace maintenance, WAL recovery, shard compaction, and backup/restore procedures.
Global Administrator Console (/ui/admin)
Administrators manage store health, vector subspace refitting, Ebbinghaus memory consolidation, shard compaction, and live MCP JSON-RPC 2.0 tool inspection from the dark-mode Global Administrator Console at http://127.0.0.1:7331/ui/admin.
Key Header Telemetry Indicators
Metric
Meaning
Operational Target
Live Nodes
Active non-tombstoned records across L3 booklets/axioms, L2 sections, L1 rules, and L0 passages.
Scales linearly; run /compact if deleted tombstones exceed 15% of total records.
HNSW
Indexed vertices inside the 4-layer Associative HNSW navigation graph (graph.vhnsw).
Should equal Live Nodes after ingestion and compaction.
LSA Dims
Active latent dimensions of the Randomized Truncated SVD model (lsa.vlsa).
Typically 32–128 (0 before initial /refit_lsa call).
VVEC Hits
Cumulative 4 KB binary vector page cache hits served from RAM without OS page faults.
> 98% hit ratio under normal WorkingSet workloads (2,048 pages = 8.0 MB cache).
Store Directory Layout & Crash-Safe Persistence
A serve_memory store directory (--dir ./data/store) maintains strict separation between append-only TOON text shards, binary paged vectors, HNSW graph topology, and atomic metadata snapshots:
Deterministic Corpus Verification (verify_artifacts)
Before promoting compiled corpora to examination laptops, compliance portals, or production serve_memory nodes, run verify_artifacts as a mandatory release gate:
verify_artifacts enforces six structural and security invariants:
Schema & Manifest Completeness: Confirms <corpus>_index.json, <corpus>_data.json, knowledge_base.html, and exports/<corpus>.md exist and parse cleanly.
Citation Provenance Coverage: Verifies every passage carries non-empty doc_id, section_id, printed_page, and valid 1-based page_index bounds.
Numerical Vector Sanity: Confirms every Truncated SVD projection row and passage concept vector matches the declared dimensionality with finite norms (zero NaN or Inf values).
Zero Unescaped Script Closers: Confirms embedded JSON blocks inside knowledge_base.html escape </script> sequences (<\/script>) to prevent DOM breakout.
100% Air-Gapped Self- Containment: Scans knowledge_base.html outside #kbdata prose to guarantee zero remote http://, https://, //, javascript:, or vbscript: references.
1. Loopback-First Binding, TLS Proxy & Production Deployment
By default, serve_memory binds exclusively to 127.0.0.1:7331. In multi-user enterprise deployments:
Keep serve_memory bound to loopback (or an isolated container network) behind a TLS-terminating reverse proxy (such as Caddy, NGINX, or Envoy) enforcing HTTPS/HSTS and mTLS or OIDC. Reference production manifests ship in Dockerfile, deploy/compose.yaml, deploy/Caddyfile, and deploy/vkbm.service.
Enable --require-auth, --secure-cookies, and --audit-log, and inject a high-entropy 256-bit secret via VKBM_AUTH_TOKEN (rather than passing secrets on shared process command lines). serve_memoryfails closed at startup if --require-auth is enabled without a configured token:
Observability & Backpressure: Scrape GET /metrics (Prometheus exposition format) for store telemetry, HTTP request/error counts, auth denials, rate-limit counters, and active connections. When active_conns reaches max_concurrent_conns (32), excess connections receive an immediate 503 Service Unavailable response without stalling the accept loop. Expensive and authentication endpoints enforce per-minute rate limiting (429 Too Many Requests), and token budgets are bounded (16..16384).
When configuring /ask or /v1/chat/completions to call an inference server (VKBM_LLM_URL or runtime POST /configure), vkbm enforces strict http:// URL validation (rejecting cleartext https:// on raw sockets to prevent accidental credential leakage—use a local TLS proxy sidecar for remote HTTPS endpoints), loopback-only targets on runtime /configure unless --allow-remote-config is explicitly enabled, CRLF/header-injection guards, Transfer-Encoding: chunked decoding, configurable --llm-timeout (default 60s), and bounded response sizes (<= 2 MB for LLM completions; <= 64 KB for inbound HTTP requests with 413 Payload Too Large).
3. Air-Gapped Operation
vkbm requires zero external network access for indexing, Randomized Truncated SVD training, VVEC v2 paging, 4-layer HNSW recall, contract gap auditing (/audit_contract), and word-for-word citation verification (/verify_citations). Only optional generative synthesis (/ask, /v1/chat/completions) or optional external Matryoshka embeddings (--embed-url) initiate outbound HTTP connections—typically to a co-located loopback vllm instance on the same air-gapped host.
Subspace Maintenance, Consolidation & Compaction
Refitting Truncated SVD (POST /refit_lsa)
Whenever corpus stewards ingest new regulatory booklets exceeding ~10% of existing store volume, trigger an online Truncated SVD refit so domain co-occurrence patterns update across all vectors:
Consistent Hot Snapshot Prep: Run POST /compact first so all pending index.wal mutations are checkpointed into index.idx, vectors.vvec, graph.vhnsw, and compacted .toon shards.
Cold Archive Copy: Stop serve_memory cleanly (SIGTERM), archive the store directory (tar -czf vkbm-store-$(date +%F).tar.gz ./data/compliance_prod), and record its SHA-256 digest.
Portable Markdown Audit Export: For human-readable archival independent of binary indices, run build_export to generate exports/<corpus>.md with full YAML frontmatter and printed-page markers.
Operations & Use Cases — Production Integration Playbooks
This guide presents four concrete operational playbooks showing how banking compliance teams, corporate legal departments, security operations centers (vsight), and autonomous AI agent pipelines deploy vkbm in production.
Use Case 1: Regulatory Handbook Q&A with vllm (Zero-Hallucination RAG)
Scenario
Bank examiners and risk officers need answers across 58 OCC Comptroller's Handbook and FFIEC IT Examination booklets (4,768 pages, 11,387 passages) running on an internal air-gapped workstation paired with vllm. Every claim must cite the exact booklet, Roman-numeral section, and printed page number.
Operational Workflow
Start serve_memory pointed at the local vllm OpenAI server:
Execute a 2-Turn Coarse-to-Fine Grounded Query (POST /ask):
BASH
curl -sS -X POST http://127.0.0.1:7331/ask \
-H "Authorization: Bearer ${VKBM_AUTH_TOKEN}" \
-H "Content-Type: application/json" \
-d '{
"question": "What are management responsibilities for testing disaster recovery resumption targets and documenting exceptions to credit underwriting policy?",
"budget": 700,
"auto_zoom": true,
"verify": true
}'
Resulting Grounded Payload:
Turn 1 loads L3 booklet nodes and L2 section centroids across FFIEC Business Continuity Management and OCC Commercial Loans, automatically expanding resumption targets <-> RPO/RTO via #syn_rpo_rto.
Turn 2 executes zoom_in into FFIEC-BCM §IV.B and OCC-CLS §II.B.1, injecting compact TOON-LD @graph rows (~470 tokens instead of 2,400+ tokens of raw chunks).
Before returning, vkbm runs its deterministic span verifier over every citation emitted by vllm and attaches per-citation VERIFIED_EXACT status badges.
Use Case 2: Legal Vendor Contract Clause Gap Analysis
Scenario
Procurement and legal counsel receive a 35-page Cloud Core Banking Master Services Agreement (MSA) from a critical SaaS provider. Before redlining, counsel needs an automated clause-by-clause gap report against OCC Bulletin 2023-17, 12 CFR Part 30, and FFIEC Information Security baselines.
Operational Workflow
Submit the vendor contract directly to POST /audit_contract:
BASH
curl -sS -X POST http://127.0.0.1:7331/audit_contract \
-H "Authorization: Bearer ${VKBM_AUTH_TOKEN}" \
-H "Content-Type: application/json" \
-d '{
"contract_name": "ApexCore Cloud Banking MSA (Draft 2026-11)",
"top_k": 3,
"contract_text": "Section 4.1 Encryption. Provider shall encrypt Customer data in transit using TLS 1.3 and at rest using AES-256, with Customer retaining logical key revocation rights.\n\nSection 7.2 Audit Scope. Customer audit rights are limited to reviewing Provider self-attested security whitepaper once every twenty-four months; on-site or regulator-led examinations are excluded.\n\nSection 9.4 Subprocessing. Provider may delegate data processing to offshore fourth-party subcontractors without prior notice to Customer."
}'
Use Case 3: SOC Runbook & Incident Grounding with vsight
Scenario
During an active security incident surfaced in vsight (for example, anomalous credential egress from a payment gateway host), Tier-2 SOC responders need immediate, citation-verified containment steps from internal Incident Response Runbooks alongside mandatory 36-hour / 72-hour regulatory notification thresholds.
Operational Workflow
Ingest internal Markdown/text SOC playbooks using "strategy": "generic_structured" alongside federal notification rules ("strategy": "federal_regulation"):
BASH
curl -sS -X POST http://127.0.0.1:7331/ingest_document \
-H "Authorization: Bearer ${VKBM_AUTH_TOKEN}" \
-H "Content-Type: application/json" \
-d '{
"doc_id": "soc-ir-04",
"document": "SOC Playbook IR-04 Credential Compromise & Token Revocation",
"strategy": "generic_structured",
"salience": 0.95,
"raw_text": "## 1. Immediate Containment (0-15 Minutes)\nPage 1\nIsolate affected host via EDR network quarantine, revoke active OIDC refresh tokens, and rotate vault transit keys.\n\n## 2. Regulatory Escalation Thresholds\nPage 2\nIf unauthorized access affects core payment settlement or customer NPI, escalate to General Counsel within 2 hours to evaluate the 36-hour federal banking agency computer-security incident notification rule (12 CFR Part 53)."
}'
When vsight triggers an alert webhook, query POST /recall with budget: 450 to attach exact runbook steps and regulatory escalation citations directly into the incident ticket in < 1 ms.
Use Case 4: Autonomous Agent Grounding via Model Context Protocol (MCP)
Scenario
An autonomous compliance engineering agent needs structured tool access to inspect store statistics, recall coarse hierarchy, zoom into specific regulatory sections, and self-verify citations before opening a pull request or compliance report.
3. Self-Audit Draft Citations Before Final Output (vkbm_verify_citations)
BASH
curl -sS -X POST http://127.0.0.1:7331/mcp \
-H "Authorization: Bearer ${VKBM_AUTH_TOKEN}" \
-H "Content-Type: application/json" \
-d '{
"jsonrpc": "2.0",
"id": 103,
"method": "tools/call",
"params": {
"name": "vkbm_verify_citations",
"arguments": {
"citations": [
{
"node_id": "n_8",
"doc_id": "occ-2023-17",
"section_id": "III.C.2",
"printed_page": "27",
"quoted_span": "Contracts for critical activities must stipulate the banking organization right to audit"
}
]
}
}
}'
Configuration & CLI Reference
This reference documents every command-line binary, runtime flag, environment variable, corpus configuration schema, .toon synset grammar, HTTP REST endpoint, OpenAI proxy endpoint, and Model Context Protocol (MCP) JSON-RPC 2.0 tool in vkbm.
1. Command-Line Binaries
vkbm — Unified Associative Memory Walkthrough
Runs an isolated temporary store walkthrough (VVEC v2 initialization, L3 axiom pinning, 4-layer handbook ingestion, Truncated SVD subspace fitting, TOON-LD recall, and zoom_in / zoom_out WorkingSet transitions).
BASH
vkbm
build_index — Structural PDF & TOC Indexer
Extracts pages via Poppler pdftotext -layout (direct argument vector execution, zero shell), runs the 2-pass Roman/decimal TOC parser and 2-regime printed-page footer state machine, and writes <corpus>_index.json.
Flag
Short
Default
Description
--corpus <id>
-c
occ
Corpus identifier resolving corpora/<id>.json.
--pdf-dir <dir>
-p
(from config)
Override source PDF directory path.
--out <path>
-o
(from config)
Override output <corpus>_index.json file path.
--max-pages <n>
-m
0(unlimited)
Optional per-PDF page cap for rapid smoke testing.
Number of latent Truncated SVD concept dimensions (16..512).
--power-iters <q>
-q
3
Subspace power iterations for singular value separation (2..6).
--in <path>
-i
(from config)
Override input <corpus>_index.json path.
--out <path>
-o
(from config)
Override output <corpus>_data.json path.
--help
-h
false
Display usage and exit.
build_page — Air-Gapped Single-File HTML Compiler
Renders <corpus>_data.json into a 100% self-contained browser search application (knowledge_base.html) with XSS-hardened single-pass JSON substitution.
Reads compiled handbook pages in canonical PDF order (deduplicating shared multi-section pages) and writes exports/<corpus>.md with YAML frontmatter and <!-- page printed="N" pdf="N" --> markers.
Audits <corpus>_index.json, <corpus>_data.json, knowledge_base.html, and exports/<corpus>.md for schema completeness, printed-page coverage, finite vector norms, and zero remote HTML references.
OpenAI-compatible model discovery endpoint (vkbm-compliance-v1, vkbm-contract-auditor).
POST /v1/chat/completions
Examiner+
OpenAI-compatible chat completion proxy with automatic TOON-LD injection and citation verification.
POST /mcp
Examiner+
Model Context Protocol JSON-RPC 2.0 endpoint (initialize, tools/list, tools/call). Enforces read-only RBAC for portal sessions (-32003 Forbidden on mutating tools).
POST /api/v1/auth/session
Public
Mints a 256-bit OS CSPRNG role-scoped session (portal, delegated, admin) and HttpOnly; SameSite=Strict cookie (requires valid auth_token when configured; rate-limited; fails closed with 503 if --require-auth has no token).
POST /api/v1/auth/logout
Public
Revokes the active server-side session token and expires the vkbm_session cookie (Max-Age=0).
GET /api/v1/self/profile
Examiner+
Returns active session role, workspace routes, and store summary.
Newly assigned @id of the pinned L3 supervisory axiom.
vkbm_stats
(none)
Live store, VVEC pager hit/fault counters, LSA dimensions, and HNSW layer counts.
Performance, Retrieval Ablation & Hardware Sizing
This guide documents empirical retrieval accuracy, printed-page citation fidelity, token-budget compression ratios, VVEC v2 quantization precision, SIMD microbenchmarks, and hardware sizing envelopes measured with bench_eval and production -prod builds.
1. Empirical Retrieval Ablation (bench_eval)
Regulatory and legal queries frequently fail under naive lexical search (because examiners ask about "resumption targets" while handbooks state "recovery time objectives") and also fail under pure dense embeddings (which blur exact statutory identifiers such as 12 CFR 30.4(c) vs. 12 CFR 53.3).
bench_eval evaluates progressive retrieval stages across golden regulatory paraphrase and exact-citation query suites:
Retrieval Configuration
Recall@5
Recall@10
Mean Reciprocal Rank (MRR)
Key Failure Mode Eliminated
1. Lexical Okapi BM25 Only
62.5%
75.0%
0.5417
Misses cross-vocabulary regulatory paraphrases (CECL <-> allowance for credit losses).
2. BM25 + Character 3–5gram Subwords
75.0%
87.5%
0.6875
Captures morphological variants (subcontracting / subcontractors), still misses disjoint synonyms.
3. + Randomized Truncated SVD (LSA Subspace)
87.5%
100.0%
0.8333
Bridges latent co-occurrence concepts across multi-booklet corpora without external GPUs.
Evaluated across 58 official OCC Comptroller's Handbook and FFIEC IT Examination booklets (4,768 PDF pages, 11,387 extracted passages):
Structural Metric
Measured Result
Engineering Mechanism
Printed-Page Footer Attribution Accuracy
98.5%
2-regime footer state machine handling Roman-numeral front matter (i..xii), booklet-prefixed footers (FFIEC IT Examination Handbook), and monotonic gap interpolation.
Multi-Line TOC Hierarchy Resolution
99.1%
2-pass lookahead Roman/letter/decimal parser (I -> A -> 1 -> a) with dot-leader stripping and running-header suppression.
Verbatim Quote Verification Precision
100.0% deterministic
Exact and whitespace-normalized substring span check directly against immutable L0.toon shards (0% false-positive hallucinated quotes).
3. Prompt Token Compression vs. Naive Flat Chunk RAG
Conventional RAG retrieves top-$K$ (K = 10..20) fixed-size 512-token chunks, injecting 5,000–10,000 tokens of repetitive boilerplate per turn. vkbm combines 2-Turn Coarse-to-Fine WorkingSet Zoom with compact TOON-LD @graph tabular serialization:
Includes #syn_* expansion, HNSW search, and TOON-LD rendering
4-Layer Associative Recall (150K nodes)
0.85 ms
0.58 ms
Warm 8.0 MBVectorPager cache
Contract Gap Audit (POST /audit_contract)
2.4 ms / clause
2.1 ms / clause
~84 ms total for a 40-clause vendor MSA
Randomized Truncated SVD Fit (12K passages, k=64)
410 ms
410 ms
Pure-CPU Halko–Martinsson–Tropp (q=3 power iterations)
Warm serve_memory/recall Throughput
1,450 QPS
2,180 QPS
Loopback HTTP keep-alive
5. Hardware & Store Sizing Reference
Because VectorPager bounds resident vector pages (--max-pages, default 2,048 pages = 8.0 MB) and .toon text records stream on demand via byte-offset seeks, vkbm runs predictably from constrained edge VMs to multi-million-passage enterprise servers:
Corpus Scale
Example Workload
VVEC v2 (256-D) Disk
.toon + HNSW Disk
Recommended --max-pages
Daemon Steady-State RAM
Small (18,000 nodes)
60 regulatory handbooks (~5,000 pages)
4.8 MB
~22 MB
2,048 (8 MB cache)
~38 MB
Medium (150,000 nodes)
500 handbooks + 1,000 vendor contracts
39 MB
~185 MB
8,192 (32 MB cache)
~115 MB
Large (1,000,000 nodes)
Enterprise policy lake + multi-year legal archive
260 MB
~1.2 GB
32,768 (128 MB cache)
~540 MB
XL (5,000,000 nodes)
National regulatory + technical library
1.30 GB
~6.1 GB
65,536 (256 MB cache)
~1.9 GB
Reproducing Benchmarks Locally
Run the deterministic benchmark binary from your compiled release directory at any time:
Pages int8 symmetric quantized vectors (260 B @ 256-D, 132 B @ 128-D) through a bounded 4 KB binary page cache (8.0 MB default RAM budget) backed by a crash-safe write-ahead log (index.wal) and atomic index snapshots (index.idx).
Scales from 18,000 handbook nodes to multi-million-passage corpora within a fixed memory envelope.
2-Turn Coarse-to-Fine WorkingSet Zoom
Loads compact L3/L2 summaries on Turn 1, identifies the highest-relevance sections, and executes zoom_in(L2 -> L1/L0) via $O(1)$ shard byte-offset seeks on Turn 2, rendering sorted TOON-LD @graph tables.
Cuts LLM prompt token consumption by 61.4%–80% compared to flat top-$K$ chunk retrieval.
Deterministic Word-for-Word Citation Verifier
Audits structured citations (@id, doc_id, section_id, printed_page, quoted_span) directly against L0 passage shards using exact and whitespace-normalized substring matching.
Eliminates fabricated section references, wrong page numbers, and paraphrased quotes masquerading as verbatim text.
Clause-by-Clause Contract Gap Auditor
Splits vendor agreements (MSAs, cloud schedules, SLAs) into normative clauses, tags deontic modalities (mandatory, prohibition, supervisory, commitment, permission), and classifies coverage as MATCHED, PARTIAL_REVIEW, or GAP_ALERT.
Evaluates a 40-clause contract in under 90 ms without requiring external GPU infrastructure.
OpenAI Proxy, MCP Server & Role-Separated Web Studio
Serves 22 REST endpoints, an OpenAI-compatible /v1/chat/completions proxy, an 8-tool JSON-RPC 2.0 MCP server (POST /mcp & --mcp-stdio), and a 4-portal browser workspace (/ui/auth, /ui/portal, /ui/delegated, /ui/admin).
Drop-in memory plane for human examiners, vllm inference gateways, vsight SOC analysts, and autonomous MCP agents.
Performance & Sizing — Empirical retrieval ablation, 98.5% printed-page accuracy across 58 booklets, token-efficiency benchmarks, and hardware sizing tables.
Part II: How-To Guide — Step-by-Step Operational Recipes
This guide provides copy-pasteable operational recipes for compiling PDF and Markdown corpora into 4-level hierarchical knowledge graphs, building hybrid BM25 + Truncated SVD indices, launching the serve_memory OpenAI and MCP server, auditing contracts for regulatory gaps, and verifying verbatim page citations.
Recipe 1: Ingest PDF Handbooks into a Deterministic Offline Corpus
Use the 5-stage offline compiler when publishing immutable handbook distributions (such as OCC Comptroller's Handbooks, FFIEC IT Examination Booklets, or internal SOP manuals) with a standalone knowledge_base.html explorer and citation-preserving Markdown export.
Prerequisites
Ensure Poppler pdftotext (>= 22.x) is installed on your PATH so build_index can extract layout-preserving page columns without invoking a system shell:
BASH
pdftotext -v
Step 1: Define the Corpus Manifest (corpora/my_policy.json)
Create a JSON configuration file inside corpora/ describing your PDF directory, header/footer rules, and curated starter queries:
Step 2: Sign In via the Role-Separated Authentication Gate
Navigate to http://127.0.0.1:7331/ui/auth to establish a scoped session (Examiner / Compliance Analyst, Corpus Steward, or Global Memory Admin) and inspect live header telemetry (Live Nodes, HNSW, LSA Dims, VVEC Hits).
Step 3: Pin a Tier-3 (L3) Supervisory Axiom
Pinned L3 axioms reserve dedicated capacity inside every WorkingSet so non-negotiable institutional mandates are never evicted during retrieval:
Step 4: Ingest Structured Documents via Strategy-Driven Parsing
Use /ui/delegated (Corpus Steward Portal) or POST /ingest_document to compile handbooks, regulations, or contracts directly into L3 -> L2 -> L1 -> L0 nodes.
BASH
curl -sS -X POST http://127.0.0.1:7331/ingest_document \
-H "Authorization: Bearer ${VKBM_AUTH_TOKEN}" \
-H "Content-Type: application/json" \
-d '{
"doc_id": "occ-2023-17",
"document": "OCC Bulletin 2023-17 Third-Party Relationships",
"strategy": "occ_ffiec_handbook",
"salience": 0.92,
"raw_text": "I. Governance and Risk Management\nPage 8\nA banking organization should adopt risk management practices commensurate with the level of risk and complexity of its third-party relationships.\n\nIII.C.2 Right to Audit and Remediation\nPage 27\nContracts for critical activities must stipulate the banking organization right to audit, review independent SOC 2 Type II reports, and enforce timely remediation."
}'
Step 5: Fit the Corpus-Adaptive Truncated SVD (LSA) Subspace
After ingesting or updating documents, refit the pure-CPU Randomized Truncated SVD model so every L3..L0 node is re-projected into the learned latent subspace:
Configure serve_memory with an upstream OpenAI-compatible inference server (such as local vllm, llama-server, or an enterprise gateway) via environment variables:
Any standard OpenAI SDK or HTTP client can now query http://127.0.0.1:7331/v1/chat/completions. vkbm intercepts the user prompt, executes 2-turn coarse-to-fine associative recall + zoom_in, injects the verified TOON-LD @graph context, calls the upstream model, and runs deterministic citation verification before returning the response:
BASH
curl -sS -X POST http://127.0.0.1:7331/v1/chat/completions \
-H "Authorization: Bearer ${VKBM_AUTH_TOKEN}" \
-H "Content-Type: application/json" \
-d '{
"model": "vkbm-grounded",
"messages": [
{
"role": "user",
"content": "What contractual audit rights are required for critical cloud vendors under OCC 2023-17?"
}
],
"temperature": 0.1
}'
Option B: Model Context Protocol (POST /mcp or --mcp-stdio)
For desktop coding assistants, compliance copilots, or autonomous orchestrators, expose the 8 grounded memory tools over JSON-RPC 2.0:
Recipe 4: Run Automated Contract Gap Analysis & Quote Verification
Step 1: Audit Draft Contract Text Against Regulatory Baselines
Submit draft agreement clauses to POST /audit_contract to classify every normative obligation (MATCHED, PARTIAL_REVIEW, or GAP_ALERT) with exact handbook citations:
BASH
curl -sS -X POST http://127.0.0.1:7331/audit_contract \
-H "Authorization: Bearer ${VKBM_AUTH_TOKEN}" \
-H "Content-Type: application/json" \
-d '{
"contract_name": "Core Ledger Cloud Hosting MSA v3",
"contract_text": "1. Audit Rights. Customer may review Provider annual SOC 2 Type II summary report upon written request, and regulatory agencies may examine critical operations.\n\n2. Subcontractors. Provider may engage fourth-party subcontractors at its sole discretion without prior customer notification.\n\n3. Incident Reporting. Provider shall notify Customer within 24 hours of confirming any unauthorized access to Customer NPI.",
"top_k": 3
}'
Step 2: Verify Every Quoted Span Word-for-Word (POST /verify_citations)
Before publishing an AI-authored compliance memo, pass its structured citations to /verify_citations to confirm that every @id, doc_id, section_id, printed_page, and quoted_span matches the L0 shard byte-for-byte:
BASH
curl -sS -X POST http://127.0.0.1:7331/verify_citations \
-H "Authorization: Bearer ${VKBM_AUTH_TOKEN}" \
-H "Content-Type: application/json" \
-d '{
"citations": [
{
"node_id": "n_8",
"doc_id": "occ-2023-17",
"section_id": "III.C.2",
"printed_page": "27",
"quoted_span": "Contracts for critical activities must stipulate the banking organization right to audit"
}
]
}'
This guide explains how compliance examiners, legal reviewers, security analysts, and technical researchers use vkbm to search multi-handbook corpora, navigate the 4-level hierarchy (Book -> Section -> Rule -> Quote), inspect confidence and printed-page anchors, and audit legal contracts.
Understanding the 4-Level Knowledge Hierarchy
Every document compiled into vkbm is structured as a 4-tier pyramid rather than disconnected flat chunks:
Anchors high-level regulatory scope and keeps non-negotiable rules permanently resident in WorkingSet memory.
L2 (tier = 2)
Section Centroid
Authoritative Table-of-Contents section (e.g., §III.C.2 — Right to Audit and Remediation) with an L2-normalized salience-weighted concept vector.
Provides fast, low-token structural orientation on Turn 1 before drilling down into verbatim clauses.
L1 (tier = 1)
Normative Rule / Obligation
Multi-sentence regulatory mandate, prohibition, or contractual commitment tagged with deontic modality (mandatory, prohibition, supervisory, commitment, permission).
Maps individual contract clauses directly against enforceable regulatory obligations.
L0 (tier = 0)
Verbatim Page-Cited Quote
Exact passage text stored in 64 MB .toon shards with immutable {doc_id, section_id, printed_page, page_index} provenance.
Supplies court- and exam-ready verbatim quotations verified word-for-word against the source page.
Signing In & Role-Scoped Workspaces
Open http://127.0.0.1:7331/ui/auth in any modern desktop browser. vkbm separates daily analytical lookup from corpus mutation and store administration across three workspaces:
Examiner / Compliance Analyst (/ui/portal) — Read-only access to Associative Recall (/recall), 2-Turn Cited Q&A (/ask), Contract Gap Audit (/audit_contract), Citation Verification (/verify_citations), and the CentralIndex Record Browser.
Corpus Steward (/ui/delegated) — Scoped privileges to ingest structured handbooks, regulations, and runbooks (/ingest_document), pin supervisory axioms (/ingest), and load .toon synset packs.
Global Memory Administrator (/ui/admin) — Full store governance including Randomized Truncated SVD refitting (/refit_lsa), Ebbinghaus consolidation (/consolidate), WAL/shard compaction (/compact), runtime vector tuning, and live MCP JSON-RPC inspection.
In the Examiner Portal (/ui/portal), select the Associative Recall & WorkingSet workspace:
Enter a natural-language query, acronym, or exact citation phrase (for example, board oversight critical vendor audit rights subcontractor cyber incident notification).
Set your Token Budget (default 500–650 tokens). vkbm automatically reserves space for pinned L3 axioms first, then loads the highest-scoring L2/L1/L0 nodes via 4-layer HNSW traversal, #syn_* synset expansion, and spreading activation (gamma = 0.25).
Inspect the TOON-LD @graph Context Pane and Record Browser:
Records are grouped deterministically by shape (axiom, booklet, section, obligation, passage) and sorted by @id.
Every L0 passage displays its doc_id, section_id, printed_page (the page number printed in the document footer), and page_index (the 1-based physical PDF page).
When answering complex multi-section regulatory questions, flat RAG systems waste thousands of tokens loading full passages from marginally relevant chapters. vkbm uses 2-Turn Coarse-to-Fine Synthesis:
Turn 1 (Coarse Orientation):vkbm retrieves compact L3 booklet headers and L2 TOC section summaries across the entire store.
Turn 2 (Selective zoom_in):vkbm identifies the highest-scoring L2 section nodes and calls zoom_in, swapping out the coarse summary for its exact L1 obligations and L0 verbatim quotes via $O(1)$ shard byte-offset seeks.
Interactive zoom_in and zoom_out Controls
You can also drive coarse-to-fine navigation manually from the UI or HTTP API:
zoom_in (POST /zoom_in): Expands an L3 booklet into its L2 sections, or expands an L2 section (such as n_3 -> OCC-2023-17 §III.C.2) into its child L1 rules and L0 verbatim quotes.
zoom_out (POST /zoom_out): Collapses child L1/L0 nodes back into their parent L2 summary, freeing WorkingSet token budget for exploring another handbook chapter.
Use the Contract Compliance Audit tab in /ui/portal to screen draft vendor agreements, cloud addenda, or internal policies against your loaded regulatory corpora:
Paste the draft agreement text and assign a descriptive document title.
Click Run Clause Audit. vkbm segments the agreement into discrete clauses, classifies each clause's deontic modality (mandatory, prohibition, supervisory, commitment, permission), and preserves legal negations (not, no, never, without, neither, nor).
Review the per-clause disposition cards:
MATCHED: Strong semantic and lexical alignment with authoritative L1/L0 regulatory rules, accompanied by exact section and printed-page citations.
PARTIAL_REVIEW: Moderate overlap or scope divergence (for example, right-to-audit restricted solely to vendor-authored executive summaries or fourth-party subcontracting lacking prior notice).
GAP_ALERT: No supporting regulatory alignment or direct contradiction with a mandatory obligation—requires legal redlining before execution.
Workflow 4: Offline Single-File Explorer (knowledge_base.html)
When operating on air-gapped examination laptops without running a local HTTP daemon, open the compiled knowledge_base.html file directly from disk (./dist/knowledge_base.html):
100% Self-Contained: All CSS, JavaScript, Truncated SVD projection matrices, and passage records are embedded inside the single HTML file with zero external fonts, scripts, or network calls.
Instant Hybrid Filtering: Type regulatory terms or click curated starter chips to rank passages across booklets, filter by handbook title or TOC section, and copy pre-formatted citations (Document, Section §X.Y, p. N) to your clipboard.
Part IV: Administrator Guide — Corpus Governance, Security & Operations
This guide covers day-2 operational administration of vkbm: persistent store anatomy, deterministic artifact verification, air-gapped deployment, Bearer authentication and SSRF controls, Truncated SVD subspace maintenance, WAL recovery, shard compaction, and backup/restore procedures.
Global Administrator Console (/ui/admin)
Administrators manage store health, vector subspace refitting, Ebbinghaus memory consolidation, shard compaction, and live MCP JSON-RPC 2.0 tool inspection from the dark-mode Global Administrator Console at http://127.0.0.1:7331/ui/admin.
Key Header Telemetry Indicators
Metric
Meaning
Operational Target
Live Nodes
Active non-tombstoned records across L3 booklets/axioms, L2 sections, L1 rules, and L0 passages.
Scales linearly; run /compact if deleted tombstones exceed 15% of total records.
HNSW
Indexed vertices inside the 4-layer Associative HNSW navigation graph (graph.vhnsw).
Should equal Live Nodes after ingestion and compaction.
LSA Dims
Active latent dimensions of the Randomized Truncated SVD model (lsa.vlsa).
Typically 32–128 (0 before initial /refit_lsa call).
VVEC Hits
Cumulative 4 KB binary vector page cache hits served from RAM without OS page faults.
> 98% hit ratio under normal WorkingSet workloads (2,048 pages = 8.0 MB cache).
Store Directory Layout & Crash-Safe Persistence
A serve_memory store directory (--dir ./data/store) maintains strict separation between append-only TOON text shards, binary paged vectors, HNSW graph topology, and atomic metadata snapshots:
Deterministic Corpus Verification (verify_artifacts)
Before promoting compiled corpora to examination laptops, compliance portals, or production serve_memory nodes, run verify_artifacts as a mandatory release gate:
verify_artifacts enforces six structural and security invariants:
Schema & Manifest Completeness: Confirms <corpus>_index.json, <corpus>_data.json, knowledge_base.html, and exports/<corpus>.md exist and parse cleanly.
Citation Provenance Coverage: Verifies every passage carries non-empty doc_id, section_id, printed_page, and valid 1-based page_index bounds.
Numerical Vector Sanity: Confirms every Truncated SVD projection row and passage concept vector matches the declared dimensionality with finite norms (zero NaN or Inf values).
Zero Unescaped Script Closers: Confirms embedded JSON blocks inside knowledge_base.html escape </script> sequences (<\/script>) to prevent DOM breakout.
100% Air-Gapped Self- Containment: Scans knowledge_base.html outside #kbdata prose to guarantee zero remote http://, https://, //, javascript:, or vbscript: references.
1. Loopback-First Binding, TLS Proxy & Production Deployment
By default, serve_memory binds exclusively to 127.0.0.1:7331. In multi-user enterprise deployments:
Keep serve_memory bound to loopback (or an isolated container network) behind a TLS-terminating reverse proxy (such as Caddy, NGINX, or Envoy) enforcing HTTPS/HSTS and mTLS or OIDC. Reference production manifests ship in Dockerfile, deploy/compose.yaml, deploy/Caddyfile, and deploy/vkbm.service.
Enable --require-auth, --secure-cookies, and --audit-log, and inject a high-entropy 256-bit secret via VKBM_AUTH_TOKEN (rather than passing secrets on shared process command lines). serve_memoryfails closed at startup if --require-auth is enabled without a configured token:
Observability & Backpressure: Scrape GET /metrics (Prometheus exposition format) for store telemetry, HTTP request/error counts, auth denials, rate-limit counters, and active connections. When active_conns reaches max_concurrent_conns (32), excess connections receive an immediate 503 Service Unavailable response without stalling the accept loop. Expensive and authentication endpoints enforce per-minute rate limiting (429 Too Many Requests), and token budgets are bounded (16..16384).
When configuring /ask or /v1/chat/completions to call an inference server (VKBM_LLM_URL or runtime POST /configure), vkbm enforces strict http:// URL validation (rejecting cleartext https:// on raw sockets to prevent accidental credential leakage—use a local TLS proxy sidecar for remote HTTPS endpoints), loopback-only targets on runtime /configure unless --allow-remote-config is explicitly enabled, CRLF/header-injection guards, Transfer-Encoding: chunked decoding, configurable --llm-timeout (default 60s), and bounded response sizes (<= 2 MB for LLM completions; <= 64 KB for inbound HTTP requests with 413 Payload Too Large).
3. Air-Gapped Operation
vkbm requires zero external network access for indexing, Randomized Truncated SVD training, VVEC v2 paging, 4-layer HNSW recall, contract gap auditing (/audit_contract), and word-for-word citation verification (/verify_citations). Only optional generative synthesis (/ask, /v1/chat/completions) or optional external Matryoshka embeddings (--embed-url) initiate outbound HTTP connections—typically to a co-located loopback vllm instance on the same air-gapped host.
Subspace Maintenance, Consolidation & Compaction
Refitting Truncated SVD (POST /refit_lsa)
Whenever corpus stewards ingest new regulatory booklets exceeding ~10% of existing store volume, trigger an online Truncated SVD refit so domain co-occurrence patterns update across all vectors:
Consistent Hot Snapshot Prep: Run POST /compact first so all pending index.wal mutations are checkpointed into index.idx, vectors.vvec, graph.vhnsw, and compacted .toon shards.
Cold Archive Copy: Stop serve_memory cleanly (SIGTERM), archive the store directory (tar -czf vkbm-store-$(date +%F).tar.gz ./data/compliance_prod), and record its SHA-256 digest.
Portable Markdown Audit Export: For human-readable archival independent of binary indices, run build_export to generate exports/<corpus>.md with full YAML frontmatter and printed-page markers.
Part V: Operations & Use Cases — Production Integration Playbooks
This guide presents four concrete operational playbooks showing how banking compliance teams, corporate legal departments, security operations centers (vsight), and autonomous AI agent pipelines deploy vkbm in production.
Use Case 1: Regulatory Handbook Q&A with vllm (Zero-Hallucination RAG)
Scenario
Bank examiners and risk officers need answers across 58 OCC Comptroller's Handbook and FFIEC IT Examination booklets (4,768 pages, 11,387 passages) running on an internal air-gapped workstation paired with vllm. Every claim must cite the exact booklet, Roman-numeral section, and printed page number.
Operational Workflow
Start serve_memory pointed at the local vllm OpenAI server:
Execute a 2-Turn Coarse-to-Fine Grounded Query (POST /ask):
BASH
curl -sS -X POST http://127.0.0.1:7331/ask \
-H "Authorization: Bearer ${VKBM_AUTH_TOKEN}" \
-H "Content-Type: application/json" \
-d '{
"question": "What are management responsibilities for testing disaster recovery resumption targets and documenting exceptions to credit underwriting policy?",
"budget": 700,
"auto_zoom": true,
"verify": true
}'
Resulting Grounded Payload:
Turn 1 loads L3 booklet nodes and L2 section centroids across FFIEC Business Continuity Management and OCC Commercial Loans, automatically expanding resumption targets <-> RPO/RTO via #syn_rpo_rto.
Turn 2 executes zoom_in into FFIEC-BCM §IV.B and OCC-CLS §II.B.1, injecting compact TOON-LD @graph rows (~470 tokens instead of 2,400+ tokens of raw chunks).
Before returning, vkbm runs its deterministic span verifier over every citation emitted by vllm and attaches per-citation VERIFIED_EXACT status badges.
Use Case 2: Legal Vendor Contract Clause Gap Analysis
Scenario
Procurement and legal counsel receive a 35-page Cloud Core Banking Master Services Agreement (MSA) from a critical SaaS provider. Before redlining, counsel needs an automated clause-by-clause gap report against OCC Bulletin 2023-17, 12 CFR Part 30, and FFIEC Information Security baselines.
Operational Workflow
Submit the vendor contract directly to POST /audit_contract:
BASH
curl -sS -X POST http://127.0.0.1:7331/audit_contract \
-H "Authorization: Bearer ${VKBM_AUTH_TOKEN}" \
-H "Content-Type: application/json" \
-d '{
"contract_name": "ApexCore Cloud Banking MSA (Draft 2026-11)",
"top_k": 3,
"contract_text": "Section 4.1 Encryption. Provider shall encrypt Customer data in transit using TLS 1.3 and at rest using AES-256, with Customer retaining logical key revocation rights.\n\nSection 7.2 Audit Scope. Customer audit rights are limited to reviewing Provider self-attested security whitepaper once every twenty-four months; on-site or regulator-led examinations are excluded.\n\nSection 9.4 Subprocessing. Provider may delegate data processing to offshore fourth-party subcontractors without prior notice to Customer."
}'
Use Case 3: SOC Runbook & Incident Grounding with vsight
Scenario
During an active security incident surfaced in vsight (for example, anomalous credential egress from a payment gateway host), Tier-2 SOC responders need immediate, citation-verified containment steps from internal Incident Response Runbooks alongside mandatory 36-hour / 72-hour regulatory notification thresholds.
Operational Workflow
Ingest internal Markdown/text SOC playbooks using "strategy": "generic_structured" alongside federal notification rules ("strategy": "federal_regulation"):
BASH
curl -sS -X POST http://127.0.0.1:7331/ingest_document \
-H "Authorization: Bearer ${VKBM_AUTH_TOKEN}" \
-H "Content-Type: application/json" \
-d '{
"doc_id": "soc-ir-04",
"document": "SOC Playbook IR-04 Credential Compromise & Token Revocation",
"strategy": "generic_structured",
"salience": 0.95,
"raw_text": "## 1. Immediate Containment (0-15 Minutes)\nPage 1\nIsolate affected host via EDR network quarantine, revoke active OIDC refresh tokens, and rotate vault transit keys.\n\n## 2. Regulatory Escalation Thresholds\nPage 2\nIf unauthorized access affects core payment settlement or customer NPI, escalate to General Counsel within 2 hours to evaluate the 36-hour federal banking agency computer-security incident notification rule (12 CFR Part 53)."
}'
When vsight triggers an alert webhook, query POST /recall with budget: 450 to attach exact runbook steps and regulatory escalation citations directly into the incident ticket in < 1 ms.
Use Case 4: Autonomous Agent Grounding via Model Context Protocol (MCP)
Scenario
An autonomous compliance engineering agent needs structured tool access to inspect store statistics, recall coarse hierarchy, zoom into specific regulatory sections, and self-verify citations before opening a pull request or compliance report.
3. Self-Audit Draft Citations Before Final Output (vkbm_verify_citations)
BASH
curl -sS -X POST http://127.0.0.1:7331/mcp \
-H "Authorization: Bearer ${VKBM_AUTH_TOKEN}" \
-H "Content-Type: application/json" \
-d '{
"jsonrpc": "2.0",
"id": 103,
"method": "tools/call",
"params": {
"name": "vkbm_verify_citations",
"arguments": {
"citations": [
{
"node_id": "n_8",
"doc_id": "occ-2023-17",
"section_id": "III.C.2",
"printed_page": "27",
"quoted_span": "Contracts for critical activities must stipulate the banking organization right to audit"
}
]
}
}
}'
Part VI: Configuration & CLI Reference
This reference documents every command-line binary, runtime flag, environment variable, corpus configuration schema, .toon synset grammar, HTTP REST endpoint, OpenAI proxy endpoint, and Model Context Protocol (MCP) JSON-RPC 2.0 tool in vkbm.
1. Command-Line Binaries
vkbm — Unified Associative Memory Walkthrough
Runs an isolated temporary store walkthrough (VVEC v2 initialization, L3 axiom pinning, 4-layer handbook ingestion, Truncated SVD subspace fitting, TOON-LD recall, and zoom_in / zoom_out WorkingSet transitions).
BASH
vkbm
build_index — Structural PDF & TOC Indexer
Extracts pages via Poppler pdftotext -layout (direct argument vector execution, zero shell), runs the 2-pass Roman/decimal TOC parser and 2-regime printed-page footer state machine, and writes <corpus>_index.json.
Flag
Short
Default
Description
--corpus <id>
-c
occ
Corpus identifier resolving corpora/<id>.json.
--pdf-dir <dir>
-p
(from config)
Override source PDF directory path.
--out <path>
-o
(from config)
Override output <corpus>_index.json file path.
--max-pages <n>
-m
0(unlimited)
Optional per-PDF page cap for rapid smoke testing.
Number of latent Truncated SVD concept dimensions (16..512).
--power-iters <q>
-q
3
Subspace power iterations for singular value separation (2..6).
--in <path>
-i
(from config)
Override input <corpus>_index.json path.
--out <path>
-o
(from config)
Override output <corpus>_data.json path.
--help
-h
false
Display usage and exit.
build_page — Air-Gapped Single-File HTML Compiler
Renders <corpus>_data.json into a 100% self-contained browser search application (knowledge_base.html) with XSS-hardened single-pass JSON substitution.
Reads compiled handbook pages in canonical PDF order (deduplicating shared multi-section pages) and writes exports/<corpus>.md with YAML frontmatter and <!-- page printed="N" pdf="N" --> markers.
Audits <corpus>_index.json, <corpus>_data.json, knowledge_base.html, and exports/<corpus>.md for schema completeness, printed-page coverage, finite vector norms, and zero remote HTML references.
OpenAI-compatible model discovery endpoint (vkbm-compliance-v1, vkbm-contract-auditor).
POST /v1/chat/completions
Examiner+
OpenAI-compatible chat completion proxy with automatic TOON-LD injection and citation verification.
POST /mcp
Examiner+
Model Context Protocol JSON-RPC 2.0 endpoint (initialize, tools/list, tools/call). Enforces read-only RBAC for portal sessions (-32003 Forbidden on mutating tools).
POST /api/v1/auth/session
Public
Mints a 256-bit OS CSPRNG role-scoped session (portal, delegated, admin) and HttpOnly; SameSite=Strict cookie (requires valid auth_token when configured; rate-limited; fails closed with 503 if --require-auth has no token).
POST /api/v1/auth/logout
Public
Revokes the active server-side session token and expires the vkbm_session cookie (Max-Age=0).
GET /api/v1/self/profile
Examiner+
Returns active session role, workspace routes, and store summary.
Newly assigned @id of the pinned L3 supervisory axiom.
vkbm_stats
(none)
Live store, VVEC pager hit/fault counters, LSA dimensions, and HNSW layer counts.
Part VII: Performance, Retrieval Ablation & Hardware Sizing
This guide documents empirical retrieval accuracy, printed-page citation fidelity, token-budget compression ratios, VVEC v2 quantization precision, SIMD microbenchmarks, and hardware sizing envelopes measured with bench_eval and production -prod builds.
1. Empirical Retrieval Ablation (bench_eval)
Regulatory and legal queries frequently fail under naive lexical search (because examiners ask about "resumption targets" while handbooks state "recovery time objectives") and also fail under pure dense embeddings (which blur exact statutory identifiers such as 12 CFR 30.4(c) vs. 12 CFR 53.3).
bench_eval evaluates progressive retrieval stages across golden regulatory paraphrase and exact-citation query suites:
Retrieval Configuration
Recall@5
Recall@10
Mean Reciprocal Rank (MRR)
Key Failure Mode Eliminated
1. Lexical Okapi BM25 Only
62.5%
75.0%
0.5417
Misses cross-vocabulary regulatory paraphrases (CECL <-> allowance for credit losses).
2. BM25 + Character 3–5gram Subwords
75.0%
87.5%
0.6875
Captures morphological variants (subcontracting / subcontractors), still misses disjoint synonyms.
3. + Randomized Truncated SVD (LSA Subspace)
87.5%
100.0%
0.8333
Bridges latent co-occurrence concepts across multi-booklet corpora without external GPUs.
Evaluated across 58 official OCC Comptroller's Handbook and FFIEC IT Examination booklets (4,768 PDF pages, 11,387 extracted passages):
Structural Metric
Measured Result
Engineering Mechanism
Printed-Page Footer Attribution Accuracy
98.5%
2-regime footer state machine handling Roman-numeral front matter (i..xii), booklet-prefixed footers (FFIEC IT Examination Handbook), and monotonic gap interpolation.
Multi-Line TOC Hierarchy Resolution
99.1%
2-pass lookahead Roman/letter/decimal parser (I -> A -> 1 -> a) with dot-leader stripping and running-header suppression.
Verbatim Quote Verification Precision
100.0% deterministic
Exact and whitespace-normalized substring span check directly against immutable L0.toon shards (0% false-positive hallucinated quotes).
3. Prompt Token Compression vs. Naive Flat Chunk RAG
Conventional RAG retrieves top-$K$ (K = 10..20) fixed-size 512-token chunks, injecting 5,000–10,000 tokens of repetitive boilerplate per turn. vkbm combines 2-Turn Coarse-to-Fine WorkingSet Zoom with compact TOON-LD @graph tabular serialization:
Includes #syn_* expansion, HNSW search, and TOON-LD rendering
4-Layer Associative Recall (150K nodes)
0.85 ms
0.58 ms
Warm 8.0 MBVectorPager cache
Contract Gap Audit (POST /audit_contract)
2.4 ms / clause
2.1 ms / clause
~84 ms total for a 40-clause vendor MSA
Randomized Truncated SVD Fit (12K passages, k=64)
410 ms
410 ms
Pure-CPU Halko–Martinsson–Tropp (q=3 power iterations)
Warm serve_memory/recall Throughput
1,450 QPS
2,180 QPS
Loopback HTTP keep-alive
5. Hardware & Store Sizing Reference
Because VectorPager bounds resident vector pages (--max-pages, default 2,048 pages = 8.0 MB) and .toon text records stream on demand via byte-offset seeks, vkbm runs predictably from constrained edge VMs to multi-million-passage enterprise servers:
Corpus Scale
Example Workload
VVEC v2 (256-D) Disk
.toon + HNSW Disk
Recommended --max-pages
Daemon Steady-State RAM
Small (18,000 nodes)
60 regulatory handbooks (~5,000 pages)
4.8 MB
~22 MB
2,048 (8 MB cache)
~38 MB
Medium (150,000 nodes)
500 handbooks + 1,000 vendor contracts
39 MB
~185 MB
8,192 (32 MB cache)
~115 MB
Large (1,000,000 nodes)
Enterprise policy lake + multi-year legal archive
260 MB
~1.2 GB
32,768 (128 MB cache)
~540 MB
XL (5,000,000 nodes)
National regulatory + technical library
1.30 GB
~6.1 GB
65,536 (256 MB cache)
~1.9 GB
Reproducing Benchmarks Locally
Run the deterministic benchmark binary from your compiled release directory at any time: