vsight is a self-contained, multi-tenant Autonomous Security Information and Event Management (SIEM), Stateful Complex Event Processing (CEP) Correlation Engine, 3-Hop Blast-Radius Telemetry Graph, 3-Model Unsupervised Online Machine Learning Anomaly Detector, Two-Tier Threat Intelligence Hub, and Closed-Loop Security Orchestration, Automation, and Response (SOAR) platform.
Distributed as a single native executable for Windows, Linux, and macOS with zero external database or message-broker dependencies, vsight ingests multi-format security telemetry (RFC 5424 Syslog, ArcSight CEF, OCSF v1.1.0, Windows Sysmon JSONL, and structured HTTP microbatches), correlates multi-stage MITRE ATT&CK kill-chains in sub-millisecond single-pass transactions, and orchestrates reversible containment across identity (vwarden), edge/WAF (vgate), host firewall, AWS, and Kubernetes planes.
Evaluates Hot IOCs inline during ingestion in O(1), scans Cold IOCs asynchronously across retained WAL history, triggers high-confidence alerts on deception honeytokens, and compares pre-compromise vs. live blast radius in nanoseconds.
How-To Guide — Step-by-step recipes for Syslog/HTTP ingestion, stateful CEP rules, STIX/IOC threat feeds, and SOAR containment playbooks.
SOC Analyst User Guide — Console operations covering alert triage, 3-part CMDB risk scoring, online ML anomaly explanations, and incident timeline investigation.
Administrator Guide — Multi-tenant retention and raw-tee digest policies, SSRF-safe webhook allowlists, WAL/snapshot durability, microbatch tuning, and RBAC/API keys.
Operations & Use Cases — Deterministic 6-stage SOC and SOAR walkthrough from credential stuffing and MFA fatigue through lateral movement and exfiltration containment.
Consolidated User Handbook — Single-page master compendium combining all End-User, How-To, Analyst, Administrator, Operations, Reference, and Performance guides.
Quickstart
Run vsight directly using the native binary for your platform (vsight.exe on Windows or ./vsight on Linux and macOS):
BASH
# Verify semantic version and active storage schema version
./vsight version
# Execute the deterministic 6-stage SOC & SOAR CLI walkthrough
./vsight walkthrough
# Run the release-readiness throughput & P99 latency benchmark gate
./vsight bench
# Launch SOC Studio & OpenAPI 3.1 server on loopback with seeded intrusion telemetry
./vsight serve 8090 ./vsight_data --dev-demo
# Launch hardened production server with RFC 5424 TCP/UDP Syslog listeners
export VSIGHT_ADMIN_PASSWORD="ChangeMe_StrongOperatorSecret_2026!"
export VSIGHT_MASTER_KEY_HEX="0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
export VSIGHT_OIDC_HMAC_SECRET="vwarden-shared-federation-signing-secret-2026"
./vsight serve 8090 ./data --host 127.0.0.1 --syslog-tcp 127.0.0.1:5514 --syslog-udp 127.0.0.1:5514
Once started, open http://127.0.0.1:8090/login in your browser to authenticate into the SOC Studio, or query http://127.0.0.1:8090/api/v1/openapi.json for the live OpenAPI 3.1.0 specification.
vsight — How-To Guide
This guide provides step-by-step, copy-pasteable operational recipes for ingesting multi-format telemetry, authoring custom stateful CEP detection rules, loading STIX 2.1 and Two-Tier IOC feeds, and configuring SSRF-hardened SOAR containment playbooks.
1. Authenticate & Obtain a Session or Scoped Collector API Key
Mutating REST API calls require either an authenticated operator session cookie paired with the X-VSight-CSRF: 1 header, or a scoped Bearer vsk_... API key.
1.1 Sign In via Operator Credentials (POST /api/v1/auth/login)
vsight normalizes heterogeneous wire streams into canonical OCSF-aligned TelemetryEvent records and commits them in bounded microbatches (<= 512 events per batch).
vsight separates indicators into two operational tiers:
hot_active — Evaluated inline in O(1) on every incoming event (ip, domain, sha256, user, process).
cold_hunt — Stored off the hot path to keep memory deterministic and used for retrospective WAL threat hunts (POST /api/v1/hunt/query), with optional automatic promotion to hot_active upon sighting.
4.1 Upsert High-Confidence Hot Indicators (POST /api/v1/intel/iocs)
4.3 Subscribe to Live TAXII 2.1 & STIX 2.1 Feeds (POST /api/v1/intel/taxii-feeds)
vsight includes an SSRF-guarded TAXII 2.1 collection poller supporting public (none) and authenticated (bearer, basic, api_key) feeds. Secrets are stored in MVCC state and redacted (***) in all API responses.
This guide walks Security Operations Center (SOC) Tier-1 Triage Analysts, Tier-2 Incident Responders, and Threat Hunters through daily investigation workflows inside the vsight Web Studio.
1. Role-Aware Workspaces & Navigation
After authenticating at /login via local operator credentials or a signed vwarden OIDC 1.0 / PKCE assertion, vsight routes operators to one of four role-aware workspace shells:
Route
Workspace Mode
Minimum Role
Primary Responsibility
/ui/portal
SOC Viewer Portal
SOC_READ_VIEWER
Read-only inspection of correlated alerts, 3-hop attack graphs, MITRE coverage, CMDB risk, and SOAR dry-run plans; mutating controls are hidden and rejected server-side (403 Forbidden).
/ui/delegated
Tier-2 Responder Console
TIER2_RESPONDER
Incident triage, SOAR containment & LIFO rollback, CMDB asset enrollment, Two-Tier IOC management, WAL threat hunts, and deception honeytokens within assigned tenant scopes.
Traditional SIEM consoles overwhelm analysts with thousands of disconnected low-level events. vsight groups correlated detections by Entity Kill-Chain by default (Group By: Entity / Kill-Chain), surfacing the highest-risk compromised hosts and identities at the top of the queue.
2.1 Reading the Real-Time Telemetry KPI Strip
Telemetry Events: Total normalized events committed to the sharded MVCC state store.
Correlated Alerts: High-confidence multi-signal alerts emitted across CEP, Hot IOC, Deception, and Online ML engines.
Peak Kill-Chain Score: Highest composite risk score across active (tenant_id, entity_id) kill-chain state machines.
Active Enforcements: Live soft or permanent SOAR containment policies active on vwarden and vgate.
Filter & Sort: Narrow the triage queue by severity (CRITICAL, HIGH & CRITICAL), MITRE ATT&CK tactic (TA0001 Initial Access through TA0010 Exfiltration), entity identifier, or free-text search across usernames, source IPs, and command lines.
Focused Entity Context Bar: Clicking any entity pill sets the global Focused Entity Context (host:web-01 by default) and exposes one-click pivot actions:
3-Hop Graph — Traverses the bounded blast-radius subgraph centered on the entity.
3-Part CMDB Risk — Computes real-time inherent, active threat, and blast-exposure risk scores.
What-If Harden — Simulates applying compensating controls or CVE patches.
SOAR Contain — Initiates a durable IncidentContainmentWorkflow for immediate quarantine.
Time-Travel Diff — Compares the entity's pre-compromise snapshot against live state.
Reset Risk — Clears residual active threat score after confirmed remediation or false-positive triage.
Cursor Alert Stream & Collective Sightings: Switch the sub-navigation bar to Live Alert & Campaign Stream (GET /api/v1/alerts/stream) to poll O(1) ring-buffer watermarks (since_seq, head_seq, lag_dropped), or Multi-Tenant Collective Threat Intel (GET /api/v1/intel/collective) to spot shared attacker IPs hitting multiple tenants (+25% score boost across 2 tenants; +50% across 3+ tenants).
Select Attack Graph (Shortcut: 2) to inspect how an intrusion propagated across source IPs, user principals, spawned processes, and internal peer hosts.
vsight evaluates dynamic risk per (tenant_id, entity_id) using a deterministic three-component model rather than static severity labels:
Inherent Asset Risk (0..35): Derived from CMDB criticality_tier (crown_jewel = 2.0x multiplier, tier1_prod = 1.5x, standard = 1.0x), trust_zone (dmz_edge, corp_internal, restricted_vault), regulated data_classifications (PCI-DSS, PII, CHD), and unmitigated known_cves.
Time-Decayed Active Threat Score (0..50): Applies exponential half-life decay (decay_half_life_sec, default 3600s) to the entity's live MITRE kill-chain composite score so stale alerts naturally cool down while active multi-stage intrusions peak immediately.
1-Hop Neighbor Blast Exposure (0..15): Measures proximity to compromised peer hosts or identities in the live telemetry graph, surfacing crown-jewel databases (host:db-01, host:vault-01) before lateral movement reaches them.
Analysts can run What-If Mitigation Simulations (POST /api/v1/cmdb/what-if) to preview the exact risk reduction achieved by network micro-segmentation, EDR deployment, or CVE patching, ranked by Return on Security Investment (ROI).
5. Unsupervised Online ML Anomaly Explanations (Tab 7: ML Baselines)
Select ML Baselines (Shortcut: 7) to inspect the per-entity streaming statistical baselines that detect zero-day behavior without offline training jobs or labeled datasets.
Interpreting the Three Online ML Models
Model
Tracked Feature Space
How Analysts Interpret Detections
1. Welford / EWMA + Page's CUSUM
Per-entity outbound bytes_out mean, variance, exponentially weighted moving average, and cumulative sum drift.
High Z-Score (>= 4.5) flags sudden large data exfiltration spikes (e.g., 50 MB+ archive transfer); rising CUSUM Drift (>= 10.0) catches slow, low-and-slow trickle exfiltration designed to stay beneath static byte thresholds.
Flags rare or unseen process spawn edges (P < 0.01 after maturity warmup), such as nginx -> sh or sh -> powershell.exe on a web-server role where nginx -> nginx-worker is normal (P > 0.95).
3. 4D Streaming Half-Space Trees (HST)
Normalized 4-tuple [cmd_entropy, log1p(bytes_out), dst_port_class, off_hours_flag] across an ensemble of randomized binary space-partitioning trees.
Produces an Isolation Anomaly Score in [0.0, 1.0]. Scores >= 0.85 indicate rare multi-dimensional outliers combining high-entropy obfuscated command lines (Base64 -enc), unusual egress ports, and large payload transfers.
O(1) Blast-Radius Time-Travel: Enter a historical WAL offset (such as pre_compromise_offset) and click Compare Snapshots (GET /api/v1/forensics/timetravel) to highlight newly compromised nodes and lateral edges introduced since the clean checkpoint.
Zero-Side-Effect Rule Backtesting: Replay historical WAL frames inside an isolated ephemeral engine (POST /api/v1/forensics/backtest) to compare alert volume under candidate thresholds (auth_fail_threshold, lateral_fanout_threshold, entropy_threshold, exfil_z_threshold) before applying changes to live tenants.
vsight — Administrator Guide
This guide covers production deployment, cryptographic secret bootstrap, multi-tenant retention and raw-tee memory policies, SSRF-hardened SOAR webhook governance, storage durability, microbatch tuning, and Role-Based Access Control (RBAC) for vsight administrators.
1. Hardened Production Bootstrap & Secrets Governance
By default, running vsight servewithout--dev-demo activates Hardened Production Mode:
Demo quick-fill buttons and sample credentials are completely omitted from /login.
Unsigned vwarden OIDC assertions are strictly rejected.
Loopback and private-network SOAR webhook URLs are blocked by the SSRF egress filter.
Required & Recommended Environment Variables
Configure secrets exclusively via environment variables (or systemd/Kubernetes secret mounts) prior to starting vsight:
High-volume SIEM deployments frequently suffer out-of-memory failures when raw log strings accumulate in RAM. vsight decouples structured correlation state from raw wire payloads using three configurable Raw-Tee Modes:
Raw-Tee Mode
RAM / MVCC Behavior
Forensic Auditability
Recommended Environment
digest_only(Default)
Computes a deterministic SHA-256 digest of every raw wire frame and discards the raw string from RAM (0 bytes raw payload bloat).
When incident responders or automated playbooks invoke external SOAR webhooks, vsight protects internal infrastructure against Server-Side Request Forgery (SSRF):
Protocol & Address Enforcement: Only https:// (and explicitly configured http:// public endpoints) are permitted. file://, gopher://, ftp://, and non-HTTP schemes are unconditionally rejected.
Blocked CIDR & Metadata Ranges:
IPv4 Loopback (127.0.0.0/8), IPv6 Loopback (::1), and IPv4-mapped IPv6 (::ffff:127.0.0.0/104).
Cryptographic Webhook Authenticity: Every outbound SOAR POST includes X-VSight-Signature-256: sha256=<hex> computed with VSIGHT_SOAR_HMAC_SECRET over the canonical payload so downstream vwarden, vgate, firewall, and EDR receivers can reject forged containment commands.
AES-256-GCM Evidence Sealing: Exported incident case dossiers and forensic attachments written into the confined exports/ directory are encrypted at rest with NIST SP 800-38D AES-256-GCM using VSIGHT_MASTER_KEY_HEX.
vsight embeds the vprevail 16-shard Copy-on-Write MVCC engine:
Co-Located Multi-Tenant Sharding: Every tenant-scoped key uses hash-tag co-location ({tenant_id|entity_id}), ensuring all sliding windows, online ML baselines, kill-chain states, and graph edges for an entity reside on a single deterministic shard for lock-free reads (~80 ns) and single-shard atomic commits.
CRC32-Verified Binary WAL: Every microbatch commit is framed with a 32-byte header and IEEE CRC32 checksum before acknowledgement.
Active-Passive / Read-Replica WAL Synchronization: Secondary vsight nodes can operate in read_replica mode (GET /api/v1/admin/ha/status) and pull incremental WAL frames from a primary node via /api/v1/admin/ha/export-wal and /api/v1/admin/ha/apply-wal:
BASH
# Export WAL delta frames starting from offset 100 on the primary node
curl -s -b cookies.txt "http://127.0.0.1:8090/api/v1/admin/ha/export-wal?from_offset=100&max_frames=500" > wal_delta.json
# Apply exported WAL delta frames onto a standby read-replica
curl -s -b replica_cookies.txt -X POST http://127.0.0.1:8091/api/v1/admin/ha/apply-wal \
-H "Content-Type: application/json" \
-H "X-VSight-CSRF: 1" \
-d @wal_delta.json
5. High-Throughput Microbatch Tuning
To maximize sustained events-per-second (EPS) while keeping P99 detection latency under 5 ms:
Batch Size (32..128 events, max 512): Grouping 64 events into a single POST /api/v1/telemetry/batch amortizes WAL frame serialization and MVCC shard snapshot publication by 15x-25x compared to single-event HTTP commits.
Flush Interval (5..15 ms): Configure edge forwarders (vgate and Syslog collectors) to flush whichever comes first: 64 queued events or 10 ms elapsed wall time.
Hot vs. Cold IOC Partitioning: Keep the inline hot_active IOC table focused on active, high-confidence (confidence >= 80) campaign indicators, and load bulk historical feeds (100k+ indicators) into cold_hunt with auto_promote_ioc_on_hit: true.
6. Multi-Tenant RBAC & Scoped API Key Lifecycle
vsight enforces a strict four-tier privilege hierarchy across both browser sessions and Bearer vsk_... API keys:
Role Constant
Rank
Permitted Operations
Tenant Boundary
COLLECTOR_INGEST
1
POST /api/v1/telemetry/{event,batch,ingest-raw} only; zero read access to alerts, graphs, or dossiers.
Pinned strictly to the API key's assigned tenant_id.
SOC_READ_VIEWER
2
Read-only access to /api/v1/overview, alerts, OCSF export, 3-hop graph, MITRE matrix, CMDB risk, ML baselines, and dry-run simulations.
Restricted to assigned tenant_id (cross-tenant queries return 403).
TIER2_RESPONDER
3
All viewer permissions plus SOAR containment/triage, playbook execution, CMDB discovery, STIX/IOC upsert, WAL threat hunts, and honeytokens.
Restricted to assigned tenant_id (peer tenants redacted in collective sightings).
SOC_LEAD_ADMIN
4
Full platform governance including custom bytecode rule promotion, per-tenant threshold configs, API key mint/revoke, listeners, WAL compaction, and global collective sightings.
Global (*) or per-tenant filterable visibility.
Auditing & Revoking Scoped API Keys
Every administrative mutation, login attempt, RBAC denial, and SOAR action is recorded in the append-only operator audit ledger (GET /api/v1/admin/audit). Revoke compromised or retired collector keys immediately via POST /api/v1/admin/api-keys/revoke:
vsight — Operations, Use Cases & Deterministic SOC Walkthrough
This guide walks through a complete, deterministic 6-stage enterprise intrusion detection and closed-loop response lifecycle—progressing from initial credential stuffing and MFA fatigue through webshell execution, LSASS credential dumping, SMB lateral movement, high-volume data exfiltration, human-in-the-loop SOAR containment, and false-positive LIFO Saga rollback.
You can execute this entire 6-stage scenario deterministically from the CLI (./vsight walkthrough) or step-by-step over the OpenAPI 3.1 REST interface as shown below.
Stage 1: Establish Normal Workload Baselines (Zero False Alerts)
Before attack traffic arrives, vsight learns normal operational patterns per (tenant_id, entity_id) and host_role across its three streaming unsupervised ML models:
Welford / EWMA + Page's CUSUM tracks normal 2 KiB PostgreSQL query payloads from host:web-01 (user:svc-nginx) to host:db-01:5432.
First-Order Markov Process Lineage learns that on host_role = "web-server", nginx -> nginx-worker is expected (P > 0.95).
4D Streaming Half-Space Trees populate normal mass profiles for low-entropy daemon command lines during business hours.
Verification: GET /api/v1/alerts?tenant_id=corp-prod returns total_alerts: 0, and vsight records the clean wal_offset checkpoint for subsequent forensic time-travel comparisons.
An external adversary (203.0.113.88) sprays invalid TOTP codes against user:bob on vwarden (5 consecutive OIDC PKCE failures within 60s), followed by a successful MFA fatigue acceptance:
Using the compromised session on host:web-01, the attacker spawns an interactive reverse shell under nginx (nginx -> sh) and executes Base64-encoded PowerShell targeting lsass.exe:
Exponential Multi-Stage Amplification: Because host:web-01 has traversed 5+ distinct MITRE tactics, its composite kill-chain risk score crosses containment_score_thresh (18.0), automatically qualifying for closed-loop SOAR containment.
When an authorized Red-Team exercise or staging script on host:app-staging-02 (user:alice, 10.0.5.44) triggers soft containment, resolving the awakeable as false_positive (or transitioning the case SLA to CLOSED_FALSE_POSITIVE) automatically runs LIFO Saga compensations—re-enabling the user in vwarden, unbanning 10.0.5.44 at vgate, and resetting active threat risk to 0.0:
Starts the hardened SOC Studio Web UI and OpenAPI 3.1 HTTP server (defaults: port = 8090, host = 127.0.0.1).
bench
benchmark
(none)
Runs the deterministic throughput and P99 latency benchmark gate across the wire normalizer, stateful CEP + online ML pipeline, and 3-hop graph BFS. Exits non-zero if any gate fails.
walkthrough
run
(none)
Executes the 6-stage deterministic SOC intrusion detection, O(1) time-travel diff, durable SOAR containment, LIFO rollback, and WAL backtest demonstration in an isolated temp store.
compact
maintenance
[data_dir] [keep_tail_frames]
Runs expired sliding-window TTL reaping, writes a point-in-time .snap checkpoint, and compacts lagged WAL segments while retaining keep_tail_frames (default 512).
openapi
(none)
[base_url]
Prints the complete OpenAPI 3.1.0 JSON specification document to stdout (default base_url: http://127.0.0.1:8090).
docs
(none)
[out_dir]
Compiles the separated End-User (dist/docs/user/) and Technical (dist/docs/technical/) HTML documentation suites with interactive vector SVG diagrams.
audit
(none)
(none)
Executes static verification rules across the workspace and prints a pass/fail compliance report.
version
-v, --version
(none)
Prints the semantic release version and active MVCC storage schema version (vsight v1.2.0 (Schema v12)).
vsight auto-detects four wire formats on TCP/UDP Syslog listeners and POST /api/v1/telemetry/ingest-raw (format_hint: auto, syslog, cef, ocsf, sysmon), in addition to structured JSON on /api/v1/telemetry/event and /api/v1/telemetry/batch:
All mutating (POST) endpoints require either Authorization: Bearer vsk_... or an authenticated HttpOnly session cookie accompanied by X-VSight-CSRF: 1.
4.1 Health, Authentication & Self-Service
Method
Endpoint
Min Role
Description
GET
/healthz
Public
Liveness, WAL offset, schema version, and HA operating mode.
GET
/api/v1/openapi.json
Public
Live OpenAPI 3.1.0 specification JSON.
POST
/api/v1/auth/login
Public
Authenticate via local password or signed vward_oidc assertion.
POST
/api/v1/auth/logout
Viewer
Revoke active session digest from MVCC session store.
GET
/api/v1/self/session
Viewer
Current principal profile, tenant scope, and offline ISO 18004 pairing QR SVG.
POST
/api/v1/delegated/onboarding
Viewer
Submit staged collector or tenant onboarding approval request.
4.2 Telemetry Ingestion, Alerts, Graph & Online ML
Method
Endpoint
Min Role
Description
POST
/api/v1/telemetry/event
Collector
Ingest and correlate a single canonical TelemetryEvent (<= 64 KiB).
POST
/api/v1/telemetry/batch
Collector
Ingest and correlate a microbatch (1..512 events) in one WAL transaction.
POST
/api/v1/telemetry/ingest-raw
Collector
Auto-normalize and ingest multi-line RFC 5424, CEF, OCSF, or Sysmon streams.
POST
/api/v1/telemetry/verify-chain
Viewer
Verify cryptographic SHA-256 hash-chain continuity across vgate Syslog lines.
Every vsight build includes an embedded deterministic self-benchmark gate (./vsight bench) that verifies wire normalization throughput, single-pass stateful CEP + 3-model Online ML + graph ingestion throughput, and lock-free 3-hop BFS blast-radius traversal latency on the local host:
Single-event transactional writes force per-event WAL framing, CRC32 checksum calculation, and MVCC copy-on-write pointer swaps. vsight achieves both high sustained EPS and sub-4-millisecond P99 correlation latency through deterministic microbatching:
Microbatch Size (events)
Flush Timer Ceiling
Sustained Ingestion (EPS)
P50 Batch Commit
P99 Batch Commit
End-to-End Alert Visibility
1 (Single-Event Endpoint)
Immediate (0 ms)
~2,400 EPS
380 us
1,450 us
< 1.5 ms
32 Events
10 ms
~17,900 EPS
510 us
1,890 us
< 12 ms
64 Events *(Recommended)*
10 ms
~22,580 EPS
786 us
2,533 us
< 13 ms
256 Events
25 ms
~26,400 EPS
2,410 us
6,820 us
< 32 ms
512 Events (Hard Cap)
50 ms
~27,900 EPS
4,650 us
11,900 us
< 62 ms
Why Latency Remains Bounded Under Attack Floods
Bounded Ring Windows (O(1) Eviction): Per-entity authentication failure and file-modification sliding windows use fixed-capacity circular timestamp buffers rather than unbounded lists.
HyperLogLog Lateral Fan-Out (O(1) Memory): Unique destination host fan-out per entity is tracked via compact HyperLogLog registers instead of growing hash sets.
Instruction-Capped Bytecode VM: Custom expr_bytecode_v2 rules execute on a stack VM with a hard 256-instruction budget and zero regular-expression backtracking, eliminating catastrophic ReDoS latency spikes.
Co-Located {tenant_id|entity_id} Sharding: All CEP windows, Online ML models, and Kill-Chain states for a host hash to the same shard, avoiding cross-shard distributed locking.
Under the default digest_only Raw-Tee Policy, vsight computes a SHA-256 chain-of-custody digest on wire arrival and never retains raw log strings in RAM. Memory usage scales strictly with active monitored entities, not raw log volume:
Hot State Footprint per 10,000 Monitored Hosts
State Namespace
Data Structure
Footprint (10,000 Hosts)
Retention / Eviction Policy
Sliding CEP Windows & HLL
Circular timestamp rings + HyperLogLog registers
~12 MB
Automatic TTL reaping (60s - 300s windows)
3-Model Online ML Ensemble
Welford/EWMA/CUSUM scalars + Markov transition table + 4D Half-Space Trees
Part I: vsight — Autonomous SIEM, Security CEP, Unsupervised Online ML & Closed-Loop SOAR Platform
Overview
vsight is a self-contained, multi-tenant Autonomous Security Information and Event Management (SIEM), Stateful Complex Event Processing (CEP) Correlation Engine, 3-Hop Blast-Radius Telemetry Graph, 3-Model Unsupervised Online Machine Learning Anomaly Detector, Two-Tier Threat Intelligence Hub, and Closed-Loop Security Orchestration, Automation, and Response (SOAR) platform.
Distributed as a single native executable for Windows, Linux, and macOS with zero external database or message-broker dependencies, vsight ingests multi-format security telemetry (RFC 5424 Syslog, ArcSight CEF, OCSF v1.1.0, Windows Sysmon JSONL, and structured HTTP microbatches), correlates multi-stage MITRE ATT&CK kill-chains in sub-millisecond single-pass transactions, and orchestrates reversible containment across identity (vwarden), edge/WAF (vgate), host firewall, AWS, and Kubernetes planes.
Evaluates Hot IOCs inline during ingestion in O(1), scans Cold IOCs asynchronously across retained WAL history, triggers high-confidence alerts on deception honeytokens, and compares pre-compromise vs. live blast radius in nanoseconds.
How-To Guide — Step-by-step recipes for Syslog/HTTP ingestion, stateful CEP rules, STIX/IOC threat feeds, and SOAR containment playbooks.
SOC Analyst User Guide — Console operations covering alert triage, 3-part CMDB risk scoring, online ML anomaly explanations, and incident timeline investigation.
Administrator Guide — Multi-tenant retention and raw-tee digest policies, SSRF-safe webhook allowlists, WAL/snapshot durability, microbatch tuning, and RBAC/API keys.
Operations & Use Cases — Deterministic 6-stage SOC and SOAR walkthrough from credential stuffing and MFA fatigue through lateral movement and exfiltration containment.
Consolidated User Handbook — Single-page master compendium combining all End-User, How-To, Analyst, Administrator, Operations, Reference, and Performance guides.
Quickstart
Run vsight directly using the native binary for your platform (vsight.exe on Windows or ./vsight on Linux and macOS):
BASH
# Verify semantic version and active storage schema version
./vsight version
# Execute the deterministic 6-stage SOC & SOAR CLI walkthrough
./vsight walkthrough
# Run the release-readiness throughput & P99 latency benchmark gate
./vsight bench
# Launch SOC Studio & OpenAPI 3.1 server on loopback with seeded intrusion telemetry
./vsight serve 8090 ./vsight_data --dev-demo
# Launch hardened production server with RFC 5424 TCP/UDP Syslog listeners
export VSIGHT_ADMIN_PASSWORD="ChangeMe_StrongOperatorSecret_2026!"
export VSIGHT_MASTER_KEY_HEX="0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
export VSIGHT_OIDC_HMAC_SECRET="vwarden-shared-federation-signing-secret-2026"
./vsight serve 8090 ./data --host 127.0.0.1 --syslog-tcp 127.0.0.1:5514 --syslog-udp 127.0.0.1:5514
Once started, open http://127.0.0.1:8090/login in your browser to authenticate into the SOC Studio, or query http://127.0.0.1:8090/api/v1/openapi.json for the live OpenAPI 3.1.0 specification.
Part II: vsight — How-To Guide
This guide provides step-by-step, copy-pasteable operational recipes for ingesting multi-format telemetry, authoring custom stateful CEP detection rules, loading STIX 2.1 and Two-Tier IOC feeds, and configuring SSRF-hardened SOAR containment playbooks.
1. Authenticate & Obtain a Session or Scoped Collector API Key
Mutating REST API calls require either an authenticated operator session cookie paired with the X-VSight-CSRF: 1 header, or a scoped Bearer vsk_... API key.
1.1 Sign In via Operator Credentials (POST /api/v1/auth/login)
vsight normalizes heterogeneous wire streams into canonical OCSF-aligned TelemetryEvent records and commits them in bounded microbatches (<= 512 events per batch).
vsight separates indicators into two operational tiers:
hot_active — Evaluated inline in O(1) on every incoming event (ip, domain, sha256, user, process).
cold_hunt — Stored off the hot path to keep memory deterministic and used for retrospective WAL threat hunts (POST /api/v1/hunt/query), with optional automatic promotion to hot_active upon sighting.
4.1 Upsert High-Confidence Hot Indicators (POST /api/v1/intel/iocs)
4.3 Subscribe to Live TAXII 2.1 & STIX 2.1 Feeds (POST /api/v1/intel/taxii-feeds)
vsight includes an SSRF-guarded TAXII 2.1 collection poller supporting public (none) and authenticated (bearer, basic, api_key) feeds. Secrets are stored in MVCC state and redacted (***) in all API responses.
This guide walks Security Operations Center (SOC) Tier-1 Triage Analysts, Tier-2 Incident Responders, and Threat Hunters through daily investigation workflows inside the vsight Web Studio.
1. Role-Aware Workspaces & Navigation
After authenticating at /login via local operator credentials or a signed vwarden OIDC 1.0 / PKCE assertion, vsight routes operators to one of four role-aware workspace shells:
Route
Workspace Mode
Minimum Role
Primary Responsibility
/ui/portal
SOC Viewer Portal
SOC_READ_VIEWER
Read-only inspection of correlated alerts, 3-hop attack graphs, MITRE coverage, CMDB risk, and SOAR dry-run plans; mutating controls are hidden and rejected server-side (403 Forbidden).
/ui/delegated
Tier-2 Responder Console
TIER2_RESPONDER
Incident triage, SOAR containment & LIFO rollback, CMDB asset enrollment, Two-Tier IOC management, WAL threat hunts, and deception honeytokens within assigned tenant scopes.
Traditional SIEM consoles overwhelm analysts with thousands of disconnected low-level events. vsight groups correlated detections by Entity Kill-Chain by default (Group By: Entity / Kill-Chain), surfacing the highest-risk compromised hosts and identities at the top of the queue.
2.1 Reading the Real-Time Telemetry KPI Strip
Telemetry Events: Total normalized events committed to the sharded MVCC state store.
Correlated Alerts: High-confidence multi-signal alerts emitted across CEP, Hot IOC, Deception, and Online ML engines.
Peak Kill-Chain Score: Highest composite risk score across active (tenant_id, entity_id) kill-chain state machines.
Active Enforcements: Live soft or permanent SOAR containment policies active on vwarden and vgate.
Filter & Sort: Narrow the triage queue by severity (CRITICAL, HIGH & CRITICAL), MITRE ATT&CK tactic (TA0001 Initial Access through TA0010 Exfiltration), entity identifier, or free-text search across usernames, source IPs, and command lines.
Focused Entity Context Bar: Clicking any entity pill sets the global Focused Entity Context (host:web-01 by default) and exposes one-click pivot actions:
3-Hop Graph — Traverses the bounded blast-radius subgraph centered on the entity.
3-Part CMDB Risk — Computes real-time inherent, active threat, and blast-exposure risk scores.
What-If Harden — Simulates applying compensating controls or CVE patches.
SOAR Contain — Initiates a durable IncidentContainmentWorkflow for immediate quarantine.
Time-Travel Diff — Compares the entity's pre-compromise snapshot against live state.
Reset Risk — Clears residual active threat score after confirmed remediation or false-positive triage.
Cursor Alert Stream & Collective Sightings: Switch the sub-navigation bar to Live Alert & Campaign Stream (GET /api/v1/alerts/stream) to poll O(1) ring-buffer watermarks (since_seq, head_seq, lag_dropped), or Multi-Tenant Collective Threat Intel (GET /api/v1/intel/collective) to spot shared attacker IPs hitting multiple tenants (+25% score boost across 2 tenants; +50% across 3+ tenants).
Select Attack Graph (Shortcut: 2) to inspect how an intrusion propagated across source IPs, user principals, spawned processes, and internal peer hosts.
vsight evaluates dynamic risk per (tenant_id, entity_id) using a deterministic three-component model rather than static severity labels:
Inherent Asset Risk (0..35): Derived from CMDB criticality_tier (crown_jewel = 2.0x multiplier, tier1_prod = 1.5x, standard = 1.0x), trust_zone (dmz_edge, corp_internal, restricted_vault), regulated data_classifications (PCI-DSS, PII, CHD), and unmitigated known_cves.
Time-Decayed Active Threat Score (0..50): Applies exponential half-life decay (decay_half_life_sec, default 3600s) to the entity's live MITRE kill-chain composite score so stale alerts naturally cool down while active multi-stage intrusions peak immediately.
1-Hop Neighbor Blast Exposure (0..15): Measures proximity to compromised peer hosts or identities in the live telemetry graph, surfacing crown-jewel databases (host:db-01, host:vault-01) before lateral movement reaches them.
Analysts can run What-If Mitigation Simulations (POST /api/v1/cmdb/what-if) to preview the exact risk reduction achieved by network micro-segmentation, EDR deployment, or CVE patching, ranked by Return on Security Investment (ROI).
5. Unsupervised Online ML Anomaly Explanations (Tab 7: ML Baselines)
Select ML Baselines (Shortcut: 7) to inspect the per-entity streaming statistical baselines that detect zero-day behavior without offline training jobs or labeled datasets.
Interpreting the Three Online ML Models
Model
Tracked Feature Space
How Analysts Interpret Detections
1. Welford / EWMA + Page's CUSUM
Per-entity outbound bytes_out mean, variance, exponentially weighted moving average, and cumulative sum drift.
High Z-Score (>= 4.5) flags sudden large data exfiltration spikes (e.g., 50 MB+ archive transfer); rising CUSUM Drift (>= 10.0) catches slow, low-and-slow trickle exfiltration designed to stay beneath static byte thresholds.
Flags rare or unseen process spawn edges (P < 0.01 after maturity warmup), such as nginx -> sh or sh -> powershell.exe on a web-server role where nginx -> nginx-worker is normal (P > 0.95).
3. 4D Streaming Half-Space Trees (HST)
Normalized 4-tuple [cmd_entropy, log1p(bytes_out), dst_port_class, off_hours_flag] across an ensemble of randomized binary space-partitioning trees.
Produces an Isolation Anomaly Score in [0.0, 1.0]. Scores >= 0.85 indicate rare multi-dimensional outliers combining high-entropy obfuscated command lines (Base64 -enc), unusual egress ports, and large payload transfers.
O(1) Blast-Radius Time-Travel: Enter a historical WAL offset (such as pre_compromise_offset) and click Compare Snapshots (GET /api/v1/forensics/timetravel) to highlight newly compromised nodes and lateral edges introduced since the clean checkpoint.
Zero-Side-Effect Rule Backtesting: Replay historical WAL frames inside an isolated ephemeral engine (POST /api/v1/forensics/backtest) to compare alert volume under candidate thresholds (auth_fail_threshold, lateral_fanout_threshold, entropy_threshold, exfil_z_threshold) before applying changes to live tenants.
Part IV: vsight — Administrator Guide
This guide covers production deployment, cryptographic secret bootstrap, multi-tenant retention and raw-tee memory policies, SSRF-hardened SOAR webhook governance, storage durability, microbatch tuning, and Role-Based Access Control (RBAC) for vsight administrators.
1. Hardened Production Bootstrap & Secrets Governance
By default, running vsight servewithout--dev-demo activates Hardened Production Mode:
Demo quick-fill buttons and sample credentials are completely omitted from /login.
Unsigned vwarden OIDC assertions are strictly rejected.
Loopback and private-network SOAR webhook URLs are blocked by the SSRF egress filter.
Required & Recommended Environment Variables
Configure secrets exclusively via environment variables (or systemd/Kubernetes secret mounts) prior to starting vsight:
High-volume SIEM deployments frequently suffer out-of-memory failures when raw log strings accumulate in RAM. vsight decouples structured correlation state from raw wire payloads using three configurable Raw-Tee Modes:
Raw-Tee Mode
RAM / MVCC Behavior
Forensic Auditability
Recommended Environment
digest_only(Default)
Computes a deterministic SHA-256 digest of every raw wire frame and discards the raw string from RAM (0 bytes raw payload bloat).
When incident responders or automated playbooks invoke external SOAR webhooks, vsight protects internal infrastructure against Server-Side Request Forgery (SSRF):
Protocol & Address Enforcement: Only https:// (and explicitly configured http:// public endpoints) are permitted. file://, gopher://, ftp://, and non-HTTP schemes are unconditionally rejected.
Blocked CIDR & Metadata Ranges:
IPv4 Loopback (127.0.0.0/8), IPv6 Loopback (::1), and IPv4-mapped IPv6 (::ffff:127.0.0.0/104).
Cryptographic Webhook Authenticity: Every outbound SOAR POST includes X-VSight-Signature-256: sha256=<hex> computed with VSIGHT_SOAR_HMAC_SECRET over the canonical payload so downstream vwarden, vgate, firewall, and EDR receivers can reject forged containment commands.
AES-256-GCM Evidence Sealing: Exported incident case dossiers and forensic attachments written into the confined exports/ directory are encrypted at rest with NIST SP 800-38D AES-256-GCM using VSIGHT_MASTER_KEY_HEX.
vsight embeds the vprevail 16-shard Copy-on-Write MVCC engine:
Co-Located Multi-Tenant Sharding: Every tenant-scoped key uses hash-tag co-location ({tenant_id|entity_id}), ensuring all sliding windows, online ML baselines, kill-chain states, and graph edges for an entity reside on a single deterministic shard for lock-free reads (~80 ns) and single-shard atomic commits.
CRC32-Verified Binary WAL: Every microbatch commit is framed with a 32-byte header and IEEE CRC32 checksum before acknowledgement.
Active-Passive / Read-Replica WAL Synchronization: Secondary vsight nodes can operate in read_replica mode (GET /api/v1/admin/ha/status) and pull incremental WAL frames from a primary node via /api/v1/admin/ha/export-wal and /api/v1/admin/ha/apply-wal:
BASH
# Export WAL delta frames starting from offset 100 on the primary node
curl -s -b cookies.txt "http://127.0.0.1:8090/api/v1/admin/ha/export-wal?from_offset=100&max_frames=500" > wal_delta.json
# Apply exported WAL delta frames onto a standby read-replica
curl -s -b replica_cookies.txt -X POST http://127.0.0.1:8091/api/v1/admin/ha/apply-wal \
-H "Content-Type: application/json" \
-H "X-VSight-CSRF: 1" \
-d @wal_delta.json
5. High-Throughput Microbatch Tuning
To maximize sustained events-per-second (EPS) while keeping P99 detection latency under 5 ms:
Batch Size (32..128 events, max 512): Grouping 64 events into a single POST /api/v1/telemetry/batch amortizes WAL frame serialization and MVCC shard snapshot publication by 15x-25x compared to single-event HTTP commits.
Flush Interval (5..15 ms): Configure edge forwarders (vgate and Syslog collectors) to flush whichever comes first: 64 queued events or 10 ms elapsed wall time.
Hot vs. Cold IOC Partitioning: Keep the inline hot_active IOC table focused on active, high-confidence (confidence >= 80) campaign indicators, and load bulk historical feeds (100k+ indicators) into cold_hunt with auto_promote_ioc_on_hit: true.
6. Multi-Tenant RBAC & Scoped API Key Lifecycle
vsight enforces a strict four-tier privilege hierarchy across both browser sessions and Bearer vsk_... API keys:
Role Constant
Rank
Permitted Operations
Tenant Boundary
COLLECTOR_INGEST
1
POST /api/v1/telemetry/{event,batch,ingest-raw} only; zero read access to alerts, graphs, or dossiers.
Pinned strictly to the API key's assigned tenant_id.
SOC_READ_VIEWER
2
Read-only access to /api/v1/overview, alerts, OCSF export, 3-hop graph, MITRE matrix, CMDB risk, ML baselines, and dry-run simulations.
Restricted to assigned tenant_id (cross-tenant queries return 403).
TIER2_RESPONDER
3
All viewer permissions plus SOAR containment/triage, playbook execution, CMDB discovery, STIX/IOC upsert, WAL threat hunts, and honeytokens.
Restricted to assigned tenant_id (peer tenants redacted in collective sightings).
SOC_LEAD_ADMIN
4
Full platform governance including custom bytecode rule promotion, per-tenant threshold configs, API key mint/revoke, listeners, WAL compaction, and global collective sightings.
Global (*) or per-tenant filterable visibility.
Auditing & Revoking Scoped API Keys
Every administrative mutation, login attempt, RBAC denial, and SOAR action is recorded in the append-only operator audit ledger (GET /api/v1/admin/audit). Revoke compromised or retired collector keys immediately via POST /api/v1/admin/api-keys/revoke:
Part V: vsight — Operations, Use Cases & Deterministic SOC Walkthrough
This guide walks through a complete, deterministic 6-stage enterprise intrusion detection and closed-loop response lifecycle—progressing from initial credential stuffing and MFA fatigue through webshell execution, LSASS credential dumping, SMB lateral movement, high-volume data exfiltration, human-in-the-loop SOAR containment, and false-positive LIFO Saga rollback.
You can execute this entire 6-stage scenario deterministically from the CLI (./vsight walkthrough) or step-by-step over the OpenAPI 3.1 REST interface as shown below.
Stage 1: Establish Normal Workload Baselines (Zero False Alerts)
Before attack traffic arrives, vsight learns normal operational patterns per (tenant_id, entity_id) and host_role across its three streaming unsupervised ML models:
Welford / EWMA + Page's CUSUM tracks normal 2 KiB PostgreSQL query payloads from host:web-01 (user:svc-nginx) to host:db-01:5432.
First-Order Markov Process Lineage learns that on host_role = "web-server", nginx -> nginx-worker is expected (P > 0.95).
4D Streaming Half-Space Trees populate normal mass profiles for low-entropy daemon command lines during business hours.
Verification: GET /api/v1/alerts?tenant_id=corp-prod returns total_alerts: 0, and vsight records the clean wal_offset checkpoint for subsequent forensic time-travel comparisons.
An external adversary (203.0.113.88) sprays invalid TOTP codes against user:bob on vwarden (5 consecutive OIDC PKCE failures within 60s), followed by a successful MFA fatigue acceptance:
Using the compromised session on host:web-01, the attacker spawns an interactive reverse shell under nginx (nginx -> sh) and executes Base64-encoded PowerShell targeting lsass.exe:
Exponential Multi-Stage Amplification: Because host:web-01 has traversed 5+ distinct MITRE tactics, its composite kill-chain risk score crosses containment_score_thresh (18.0), automatically qualifying for closed-loop SOAR containment.
When an authorized Red-Team exercise or staging script on host:app-staging-02 (user:alice, 10.0.5.44) triggers soft containment, resolving the awakeable as false_positive (or transitioning the case SLA to CLOSED_FALSE_POSITIVE) automatically runs LIFO Saga compensations—re-enabling the user in vwarden, unbanning 10.0.5.44 at vgate, and resetting active threat risk to 0.0:
Starts the hardened SOC Studio Web UI and OpenAPI 3.1 HTTP server (defaults: port = 8090, host = 127.0.0.1).
bench
benchmark
(none)
Runs the deterministic throughput and P99 latency benchmark gate across the wire normalizer, stateful CEP + online ML pipeline, and 3-hop graph BFS. Exits non-zero if any gate fails.
walkthrough
run
(none)
Executes the 6-stage deterministic SOC intrusion detection, O(1) time-travel diff, durable SOAR containment, LIFO rollback, and WAL backtest demonstration in an isolated temp store.
compact
maintenance
[data_dir] [keep_tail_frames]
Runs expired sliding-window TTL reaping, writes a point-in-time .snap checkpoint, and compacts lagged WAL segments while retaining keep_tail_frames (default 512).
openapi
(none)
[base_url]
Prints the complete OpenAPI 3.1.0 JSON specification document to stdout (default base_url: http://127.0.0.1:8090).
docs
(none)
[out_dir]
Compiles the separated End-User (dist/docs/user/) and Technical (dist/docs/technical/) HTML documentation suites with interactive vector SVG diagrams.
audit
(none)
(none)
Executes static verification rules across the workspace and prints a pass/fail compliance report.
version
-v, --version
(none)
Prints the semantic release version and active MVCC storage schema version (vsight v1.2.0 (Schema v12)).
vsight auto-detects four wire formats on TCP/UDP Syslog listeners and POST /api/v1/telemetry/ingest-raw (format_hint: auto, syslog, cef, ocsf, sysmon), in addition to structured JSON on /api/v1/telemetry/event and /api/v1/telemetry/batch:
All mutating (POST) endpoints require either Authorization: Bearer vsk_... or an authenticated HttpOnly session cookie accompanied by X-VSight-CSRF: 1.
4.1 Health, Authentication & Self-Service
Method
Endpoint
Min Role
Description
GET
/healthz
Public
Liveness, WAL offset, schema version, and HA operating mode.
GET
/api/v1/openapi.json
Public
Live OpenAPI 3.1.0 specification JSON.
POST
/api/v1/auth/login
Public
Authenticate via local password or signed vward_oidc assertion.
POST
/api/v1/auth/logout
Viewer
Revoke active session digest from MVCC session store.
GET
/api/v1/self/session
Viewer
Current principal profile, tenant scope, and offline ISO 18004 pairing QR SVG.
POST
/api/v1/delegated/onboarding
Viewer
Submit staged collector or tenant onboarding approval request.
4.2 Telemetry Ingestion, Alerts, Graph & Online ML
Method
Endpoint
Min Role
Description
POST
/api/v1/telemetry/event
Collector
Ingest and correlate a single canonical TelemetryEvent (<= 64 KiB).
POST
/api/v1/telemetry/batch
Collector
Ingest and correlate a microbatch (1..512 events) in one WAL transaction.
POST
/api/v1/telemetry/ingest-raw
Collector
Auto-normalize and ingest multi-line RFC 5424, CEF, OCSF, or Sysmon streams.
POST
/api/v1/telemetry/verify-chain
Viewer
Verify cryptographic SHA-256 hash-chain continuity across vgate Syslog lines.
Every vsight build includes an embedded deterministic self-benchmark gate (./vsight bench) that verifies wire normalization throughput, single-pass stateful CEP + 3-model Online ML + graph ingestion throughput, and lock-free 3-hop BFS blast-radius traversal latency on the local host:
Single-event transactional writes force per-event WAL framing, CRC32 checksum calculation, and MVCC copy-on-write pointer swaps. vsight achieves both high sustained EPS and sub-4-millisecond P99 correlation latency through deterministic microbatching:
Microbatch Size (events)
Flush Timer Ceiling
Sustained Ingestion (EPS)
P50 Batch Commit
P99 Batch Commit
End-to-End Alert Visibility
1 (Single-Event Endpoint)
Immediate (0 ms)
~2,400 EPS
380 us
1,450 us
< 1.5 ms
32 Events
10 ms
~17,900 EPS
510 us
1,890 us
< 12 ms
64 Events *(Recommended)*
10 ms
~22,580 EPS
786 us
2,533 us
< 13 ms
256 Events
25 ms
~26,400 EPS
2,410 us
6,820 us
< 32 ms
512 Events (Hard Cap)
50 ms
~27,900 EPS
4,650 us
11,900 us
< 62 ms
Why Latency Remains Bounded Under Attack Floods
Bounded Ring Windows (O(1) Eviction): Per-entity authentication failure and file-modification sliding windows use fixed-capacity circular timestamp buffers rather than unbounded lists.
HyperLogLog Lateral Fan-Out (O(1) Memory): Unique destination host fan-out per entity is tracked via compact HyperLogLog registers instead of growing hash sets.
Instruction-Capped Bytecode VM: Custom expr_bytecode_v2 rules execute on a stack VM with a hard 256-instruction budget and zero regular-expression backtracking, eliminating catastrophic ReDoS latency spikes.
Co-Located {tenant_id|entity_id} Sharding: All CEP windows, Online ML models, and Kill-Chain states for a host hash to the same shard, avoiding cross-shard distributed locking.
Under the default digest_only Raw-Tee Policy, vsight computes a SHA-256 chain-of-custody digest on wire arrival and never retains raw log strings in RAM. Memory usage scales strictly with active monitored entities, not raw log volume:
Hot State Footprint per 10,000 Monitored Hosts
State Namespace
Data Structure
Footprint (10,000 Hosts)
Retention / Eviction Policy
Sliding CEP Windows & HLL
Circular timestamp rings + HyperLogLog registers
~12 MB
Automatic TTL reaping (60s - 300s windows)
3-Model Online ML Ensemble
Welford/EWMA/CUSUM scalars + Markov transition table + 4D Half-Space Trees