HD: Operators & Admins Manual

This manual is about running an instance. Installing one is install/ADMIN_GUIDE.md, which covers host requirements, sizing, the cluster definition, and the Ansible and Kubernetes paths. The invariants below — the cpuset split, the core counts — are the ones the installer enforces, so a deployment it produced already satisfies them.

HD is a high-performance, strictly segmented Data Management System built for massive-scale analytics and isolated tenancy. It heavily leverages C++ native processing (DuckDB, DPDK, NORM multicast) paired with Go-based control planes (DMS-Core, Proxy) for orchestration.

High-Level Architecture

The system uses three distinct planes isolated at the network level:

1. Control Plane (Port 6986): Over-the-wire HTTPS mTLS APIs for node registration, policy distribution, and query telemetry orchestration. Records never pass through this network.
2. Data Plane (Port 6987): Unidirectional, ultra-fast binary streaming using the FermiHDI HD Wire Protocol (FHDWP). Data flows natively from Storage Nodes back to the requestor over TCP.
3. Internal DMS / CA Plane (Port 6985 / 6989): Secure channels exclusively connecting the Datastore Management System components to bootstrap new nodes and provision PKI identities.

Core Component Lifecycles

Datastore Management System (DMS)

The DMS is the single source of truth for the HD Instance. * DMS-Core: The orchestration engine. It distributes dataset schemas and routing topologies to proxies. * DMS-CA: Issues cryptographic certificates for new nodes joining the cluster. * DMS-WebUX: Provides the operator UI for dataset administration. * OTEL Collector: Every service in the instance reports traces, metrics and logs to it. It runs under DMS-Core's management: DMS-Core renders its configuration (the exporters an administrator adds on the OTEL Collector page and, only while the administrator has opted in to sharing and the deployment is registered, the pipeline to FermiHDI) and the collector restarts on it. Ports 4317 (gRPC) and 4318 (HTTP) on the DMS host. * Crash vault: Every crash report a service posts (the engine nodes through Crashpad, the DMS and Proxy through the Go runtime, the WebUX through Node) is held on the DMS host, symbolicated against the symbol sets shipped beside the engine images (<image>-symbols:<tag>), and forwarded to FermiHDI only as an operator decides on the WebUX Telemetry page. The nodes reach it on port 3001 of the DMS host; the WebUX on 3002.

HD Proxy (Query Gateways)

Acts as the ingress gateway for the Control Plane. * Evaluates Attribute/Policy-Based Access Control (PBAC) using Open Policy Agent (OPA). * Dynamically rewrites queries with binary bitmask filters. * Forwards authorized payloads to Storage Cluster Controllers.

Storage Cluster Controllers (SCC)

SCCs orchestrate queries across their domain. * They operate as Fan-Out nodes, broadcasting queries to their underlying Storage Nodes (SNs). * SCCs ingest raw records, hashing and distributing them identically across their SNs over the Data Plane (TCP, port 6987). * NORM (NACK-Oriented Reliable Multicast, UDP port 6988, group 239.255.0.1) is implemented in libfermihdi_transport and enabled by default there, but the SCC deliberately disables it (norm_enabled = false) and fans out over TCP instead. Treat 6988 as reserved: nothing in 2.0.x sends on it, so a firewall rule permitting it is not required for a working deployment.

Storage Nodes (SN)

The raw engine layer, executing the native C++ retrieval pipelines. * They map raw FHDWP payloads directly to DuckDB Apache Arrow buffers. * Once data is found, they connect directly to the Client/Target Data Port to stream the response, bypassing the Proxy completely.

BloomRF Filter Node (optional)

An SN-shaped engine role that keeps a filter per batch instead of the records, so the SCC can leave batches unread. * It registers as node type HD_BLOOMRF inside one Storage Cluster, receives the SCC's ingest broadcast and verifies every batch's hash with the SCC before its filter becomes usable. * The SCC learns of it once, when its data plane starts, and the DMS holds the SCC's targets until every provisioned SN and filter node is ready. Adding or removing one on a running cluster therefore ends with an SCC restart. * Its filters persist to the node's local storage and are reloaded at start, so a restart keeps every batch it held; the store is bounded by a retention count. Losing a filter costs selectivity, never answers. * It needs no licence slot of its own. The HD_BLOOMRF element is taken by each SCC that runs the filter in active mode and returned when it leaves it; an SCC refused a slot runs the filter passive and logs why. * The mode — off, passive, active — is a property of the HD instance, set on the deployment's card in the WebUX and pushed to every SCC together, and reported on every query. The filter node page has the modes, sizing and metrics; the WebUX walk-through has the pages.

Message Bus Ingester

Reads one source — a Kafka, RabbitMQ or MQTT broker, or files from a directory or an S3-compatible store — and streams batches to an SCC. * It registers with the DMS like every node and is handed the record size, the SCC's address and the hash election. * A file source removes each file only once the SCC has acknowledged its records, so a directory that stops draining points at the storage cluster, not the ingester. The ingestion page covers formats and batching.

Query services

hd_sql, hd_graphql, hd_ipflow_graphql and hd_mcp are stateless services outside the storage cluster. * Each reaches the instance through a Proxy it is configured with, fetches the dataset schema from the DMS or the Proxy at start-up, and rebuilds its view when the schema changes. * Each listens for query results itself on the data plane, so every one of them needs the data-plane subnet declared and a host the Storage Nodes can reach — the section below is about exactly that. * None has access control of its own beyond a password on the SQL server's PostgreSQL listener; all belong behind the authenticating front door, like the Proxy. The front doors page is the contract for that door and for the DMS's.

Query Result Return Path

Query results do not travel back the way the query came. The SN opens a new outbound FHDWP connection to a target the caller nominated, so that target must be an address the SN can reach and a port something is listening on. Getting this wrong fails silently: the query is accepted, the SN streams to an address nobody is listening on, and the caller simply waits.

How the target is chosen

Every query carries a required target array (schemas/hd_query_schema.json), and each entry pairs an address with a required port_number. There is no default in the schema — the caller must state one.

The Proxy fills the gap for callers that omit it, in this order:

1. the caller's own target_dps[0].port_number, validated 1–65535;
2. the X-Data-Return-Port request header;
3. 6987, the FHDWP data-plane port.

The SN uses target_dps[0] only. The schema models target as an array, which reads like fan-out to several destinations; the SN streams to the first entry and ignores the rest.

Configuring it on the query services

hd_mcp, hd_graphql and hd_ipflow_graphql each issue queries and listen for the results themselves, so they advertise their own address:

Variable	Default	Purpose
FERMIHDI_DATA_RETURN_HOST	(discovery, then hostname)	Address advertised to SNs. Set it when discovery cannot pick the right interface.
FERMIHDI_DATA_RETURN_PORT	6987	Port advertised to SNs, and the port the service listens on. Out-of-range or unparseable values fall back to the default silently.
FERMIHDI_DATA_PLANE_SUBNET	10.150.0.0/24	CIDR used to pick the data-plane interface when the host is not set explicitly.

Resolution order for the host is: FERMIHDI_DATA_RETURN_HOST if set, otherwise the first local interface address inside FERMIHDI_DATA_PLANE_SUBNET, otherwise the container hostname. Discovery exists because DNS is unreliable across multiple container networks — a service name can resolve to the front-end network rather than the data plane — but an explicit host always wins over it.

Version note. Before 2.0.3, hd_mcp and hd_graphql ran subnet discovery unconditionally and overwrote FERMIHDI_DATA_RETURN_HOST, so on any host with an address in the data-plane subnet the variable silently did nothing. Only hd_ipflow_graphql honoured it. If you are on an earlier build and the setting appears to be ignored, that is why; steer the choice with FERMIHDI_DATA_PLANE_SUBNET instead, or upgrade.

Operating Principles

• Immutable Deployments: The system is intended to be run completely inside container orchestration systems (Docker Compose or Kubernetes). Do not mutate host networking.
• Certificate Lifecycle: every certificate in an HD instance renews itself without a restart; the rules are in Certificate lifecycle and rules below.
• Observability: All nodes emit OpenTelemetry metrics and traces to a shared OTEL Collector. Node-level health and Prom HTTP endpoints are exposed on port 8080 by default. See observability.md for exactly what each node type emits — it is not uniform across the fleet.

Certificate lifecycle and rules

Every process in an HD instance holds a certificate the DMS-CA signed, and every one of them renews it without a restart. The rules:

• A node certificate lives 30 days (CA_LEAF_VALIDITY_DAYS on the DMS-CA), never past the ICA's own expiry.
• A node renews its own certificate inside the last 10 days (FERMIHDI_CERT_RENEW_DAYS on the node): when it starts with an existing identity, and once a day after. It presents its current certificate to the DMS over mTLS and receives a new one for a new key. No token, no administrator.
• A renewed certificate is taken in place. The node's command listener swaps its certificate and trust without stopping; its outgoing connections use the new material from the next one opened; the engine's gRPC servers and channels watch the identity files and follow them; the Proxy's transports resolve the certificate at every handshake. Connections already open keep the material they were opened with until they reconnect.
• An expired certificate cannot renew itself. A node kept off for longer than a certificate's lifetime registers again, with a token or an approval, exactly as a new node.
• The DMS's own services (DMS-Core, the OPA engine, the WebUX) hold a 365-day certificate (CA_INTERNAL_VALIDITY_DAYS) and are re-enrolled by the DMS-CA whenever the ICA changes.
• The ICA renews against the same key. Renew or Replace License on the WebUX submits a CSR for the key the DMS-CA already holds, so every certificate it signed stays valid; the renewed ICA reaches every node in the next configuration push and is pinned in place. A licence that changes the key goes through the same page and replaces the ICA outright, after which every node's certificate is renewed against the new ICA.
• Trust is pinned to the issuing ICA only. A node verifies the DMS, and the DMS verifies a node, against the ICA that signed them, never the FermiHDI root on its own; the identity files a node keeps are its certificate, its key and that ICA.
• The WebUX shows each node's expiry, amber inside the renewal window and red once passed.

Container Deployment Model

All HD components (Storage Node, Storage Cluster Controller, HD Proxy, Message Bus Ingester, and DMS components) must always be deployed as containers, even when running as the only workload on a dedicated bare-metal server. This is a deliberate design principle to align with modern data-centre management practices and orchestration systems (Docker Compose, Kubernetes, Canonical Juju, etc.).

DPDK / SPDK Components: cpuset Split Invariant

Components that use kernel-bypass hardware (Storage Nodes and Storage Cluster Controllers with DPDK networking or SPDK storage) have an additional operational invariant that must be followed on every deployment:

The container cpuset and the MASTER_CPUSET environment variable must reference disjoint sets of CPUs. DPDK-pinned cores must never appear in the container cpuset.

Why this matters: The container runtime shim (containerd-shim on Docker, CRI-O shim on Kubernetes) runs in the host scheduler but inherits the container's cpuset cgroup. If a DPDK core appears in the container cpuset, the shim will be scheduled on that core, evicting DPDK poll-mode threads. This was measured at over 300,000 shim evictions per 90-second window in HD profiling — sufficient to cause measurable write-path latency spikes and query P99 regressions. The isolcpus kernel parameter alone does not prevent this, because an explicit cgroup cpuset assignment overrides isolcpus.

Correct split:

Field	Must contain
cpuset (Docker Compose / K8s pod spec)	Overflow / management core(s) only
MASTER_CPUSET (env var)	All DPDK/SPDK-pinned cores — excluded from cpuset

See hd_sn/docs/user_guide.md — Step 3a for full examples, profiling evidence, and Kubernetes guidance.

Deployment Planning: CPU Core Count Requirements

Use the following table when sizing a host for a HD Storage Cluster. All figures are logical CPU (thread) counts, based on profiling data from RC12 benchmark runs. These figures assume SMT (Hyper-Threading) is enabled.

Role	DPDK Cores (MASTER_CPUSET)	Overflow Cores (cpuset)	Isolation requirement
Storage Node (SN)	10 minimum (8 with SHARE_KV_WITH_STORAGE)	2	Overflow must be on a different physical core from DPDK cores
Storage Cluster Controller (SCC)	6 minimum (7 + N_SNs recommended)	1	SMT sibling acceptable on space-constrained hosts (see note below)
Query Proxy (HD Proxy)	1–2	1	No strict isolation — proxy is not DPDK-intensive

The SN figure is a hard floor, not a recommendation. MASTER_CPUSET is divided between five buckets — transport, storage, filter, hash and kv — each with a floor of 2 logical CPUs. Below 10 (or 8 when SHARE_KV_WITH_STORAGE=true lets kv alias storage) FermiController::Init() throws and the Storage Node does not start. Surplus above the floor is weighted, with the filter taking the largest share: at a MASTER_CPUSET of 28 the split is transport 5, storage 5, filter 10, hash 4, kv 4. See tests/sandbox_dms/sn_resource_recommendations.md, which is the source of truth for this.

Production sizing formula. Once above the floor, size from the workload rather than the minimum: 6 + (3 × active_drives) cores per SN and 7 + N_SNs cores for the SCC. tests/sandbox_dms/fermihdi_sizing.py computes it. The sandbox's own auto-allocation is deliberately more generous again and is not a production guide.

Minimum host sizes for a single Storage Cluster (1 SCC + 3 SNs):

Scenario	Logical CPUs
Absolute floor, SHARE_KV_WITH_STORAGE=true	36 — OS 2 + SCC 10 + 3 × 8
Absolute floor, kv on its own cores	42 — OS 2 + SCC 10 + 3 × 10
Production, sized from the formula above	46+
Production recommended (per-SN NUMA isolation)	46+ across two NUMA sockets

These are floors, not targets: a host sized exactly to one runs every bucket at its 2-core minimum with no surplus to weight. An earlier version of this table gave 16 logical CPUs for 1 SCC + 3 SNs, which cannot start — three SNs alone need 30 at the floor.

Fewer Storage Nodes fit a smaller host. The floor is per-SN, so 1 SCC + 1 SN needs 20 logical CPUs (2 + 8 + 10), and 1 SCC + 2 SNs needs 30. A host that cannot meet the per-service minimum falls back to a shared pool, where every service's cpuset is the whole available set and the SN's auto_distribute picks its threads from it.

SCC on fully-allocated hosts: When no spare physical core is available for a dedicated SCC overflow (as in the 16-CPU sandbox), the SCC may safely operate with cpuset == MASTER_CPUSET. The SCC reader thread is not a DPDK PMD spin-poller; it blocks between batches. Profiling shows the resulting scheduler events on SCC DPDK cores are swapper idle handoffs, not active preemption. This is benign. On hosts with ≥ 18 logical CPUs, allocate a dedicated overflow core to the SCC.

Disaster Recovery & Troubleshooting

Node Quarantine

If an SN begins throwing Bad Batch Notice (BBN) Control Messages, quarantine the node via the WebUX. The SCC will dynamically re-route queries to healthy replicas within the Storage Cluster.

A filter node that excludes nothing

Check the SCC's bloomrf_lookup log line for the query. mode: off means the SCC's data plane started before the filter node was ready or the deployment record names off: restart the SCC once the node reports ready. no_filter equal to candidates means the node holds no verified filter for the window — it started after those batches, evicted them, or its verification stream to the SCC is down (fermihdi_bloomrf_filters_pending climbing). A workload whose predicates fall on the batch-id column, use !=, or name values present in every batch is not a fault.

Ingest that never starts on a cluster with a filter node provisioned

The DMS holds the SCC's relay list until every provisioned Storage Node and filter node in the cluster reports ready. Start the filter node, or decommission it in the DMS; do not leave a provisioned node unrun.

Re-establishing Data Plane Sync

If an SN falls behind due to network partition on the NORM interface (UDP 6988), reboot the SN. The SCC will trigger a cold-boot sync event over standard TCP fallback on the Control Plane to ensure data integrity before marking the node Ready.