docs.fermihdi.io — system_operators_manual.md

/ › hd › 2.0.3 › md

Download system_operators_manual.mdView raw

HD: Operators & Admins Manual

This manual is about running an instance. Installing one is install/ADMIN_GUIDE.md, which covers host requirements, sizing, the cluster definition, and the Ansible and Kubernetes paths. The invariants below — the cpuset split, the core counts — are the ones the installer enforces, so a deployment it produced already satisfies them.

HD is a high-performance, strictly segmented Data Management System built for massive-scale analytics and isolated tenancy. It heavily leverages C++ native processing (DuckDB, DPDK, NORM multicast) paired with Go-based control planes (DMS-Core, Proxy) for orchestration.

High-Level Architecture

The system uses three distinct planes isolated at the network level:

  1. Control Plane (Port 6986): Over-the-wire HTTPS mTLS APIs for node registration, policy distribution, and query telemetry orchestration. Records never pass through this network.
  2. Data Plane (Port 6987): Unidirectional, ultra-fast binary streaming using the FermiHDI HD Wire Protocol (FHDWP). Data flows natively from Storage Nodes back to the requestor over TCP.
  3. Internal DMS / CA Plane (Port 6985 / 6989): Secure channels exclusively connecting the Datastore Management System components to bootstrap new nodes and provision PKI identities.

Core Component Lifecycles

Datastore Management System (DMS)

The DMS is the single source of truth for the HD Instance.

HD Proxy (Query Gateways)

Acts as the ingress gateway for the Control Plane.

Storage Cluster Controllers (SCC)

SCCs orchestrate queries across their domain.

Storage Nodes (SN)

The raw engine layer, executing the native C++ retrieval pipelines.

Query Result Return Path

Query results do not travel back the way the query came. The SN opens a new outbound FHDWP connection to a target the caller nominated, so that target must be an address the SN can reach and a port something is listening on. Getting this wrong fails silently: the query is accepted, the SN streams to an address nobody is listening on, and the caller simply waits.

How the target is chosen

Every query carries a required target array (schemas/hd_query_schema.json), and each entry pairs an address with a required port_number. There is no default in the schema — the caller must state one.

The Proxy fills the gap for callers that omit it, in this order:

  1. the caller's own target_dps[0].port_number, validated 1–65535;
  2. the X-Data-Return-Port request header;
  3. 6987, the FHDWP data-plane port.

The SN uses target_dps[0] only. The schema models target as an array, which reads like fan-out to several destinations; the SN streams to the first entry and ignores the rest.

Configuring it on the query services

hd_mcp, hd_graphql and hd_ipflow_graphql each issue queries and listen for the results themselves, so they advertise their own address:

Variable Default Purpose
FERMIHDI_DATA_RETURN_HOST (discovery, then hostname) Address advertised to SNs. Set it when discovery cannot pick the right interface.
FERMIHDI_DATA_RETURN_PORT 6987 Port advertised to SNs, and the port the service listens on. Out-of-range or unparseable values fall back to the default silently.
FERMIHDI_DATA_PLANE_SUBNET 10.150.0.0/24 CIDR used to pick the data-plane interface when the host is not set explicitly.

Resolution order for the host is: FERMIHDI_DATA_RETURN_HOST if set, otherwise the first local interface address inside FERMIHDI_DATA_PLANE_SUBNET, otherwise the container hostname. Discovery exists because DNS is unreliable across multiple container networks — a service name can resolve to the front-end network rather than the data plane — but an explicit host always wins over it.

Version note. Before 2.0.3, hd_mcp and hd_graphql ran subnet discovery unconditionally and overwrote FERMIHDI_DATA_RETURN_HOST, so on any host with an address in the data-plane subnet the variable silently did nothing. Only hd_ipflow_graphql honoured it. If you are on an earlier build and the setting appears to be ignored, that is why; steer the choice with FERMIHDI_DATA_PLANE_SUBNET instead, or upgrade.

Operating Principles

Container Deployment Model

All HD components (Storage Node, Storage Cluster Controller, HD Proxy, Message Bus Ingester, and DMS components) must always be deployed as containers, even when running as the only workload on a dedicated bare-metal server. This is a deliberate design principle to align with modern data-centre management practices and orchestration systems (Docker Compose, Kubernetes, Canonical Juju, etc.).

DPDK / SPDK Components: cpuset Split Invariant

Components that use kernel-bypass hardware (Storage Nodes and Storage Cluster Controllers with DPDK networking or SPDK storage) have an additional operational invariant that must be followed on every deployment:

The container cpuset and the MASTER_CPUSET environment variable must reference disjoint sets of CPUs. DPDK-pinned cores must never appear in the container cpuset.

Why this matters: The container runtime shim (containerd-shim on Docker, CRI-O shim on Kubernetes) runs in the host scheduler but inherits the container's cpuset cgroup. If a DPDK core appears in the container cpuset, the shim will be scheduled on that core, evicting DPDK poll-mode threads. This was measured at over 300,000 shim evictions per 90-second window in HD profiling — sufficient to cause measurable write-path latency spikes and query P99 regressions. The isolcpus kernel parameter alone does not prevent this, because an explicit cgroup cpuset assignment overrides isolcpus.

Correct split:

Field Must contain
cpuset (Docker Compose / K8s pod spec) Overflow / management core(s) only
MASTER_CPUSET (env var) All DPDK/SPDK-pinned cores — excluded from cpuset

See hd_sn/docs/user_guide.md — Step 3a for full examples, profiling evidence, and Kubernetes guidance.

Deployment Planning: CPU Core Count Requirements

Use the following table when sizing a host for a HD Storage Cluster. All figures are logical CPU (thread) counts, based on profiling data from RC12 benchmark runs. These figures assume SMT (Hyper-Threading) is enabled.

Role DPDK Cores (MASTER_CPUSET) Overflow Cores (cpuset) Isolation requirement
Storage Node (SN) 4 2 Overflow must be on a different physical core from DPDK cores
Storage Cluster Controller (SCC) 2 1 SMT sibling acceptable on space-constrained hosts (see note below)
Query Proxy (HD Proxy) 1–2 1 No strict isolation — proxy is not DPDK-intensive

Minimum host sizes for a single Storage Cluster (1 SCC + 3 SNs):

Scenario Logical CPUs
Sandbox / development 16 (8 physical cores, SMT×2)
Production minimum (clean overflow isolation) 20+
Production recommended (per-SN NUMA isolation) 32+ (dual NUMA socket)

SCC on fully-allocated hosts: When no spare physical core is available for a dedicated SCC overflow (as in the 16-CPU sandbox), the SCC may safely operate with cpuset == MASTER_CPUSET. The SCC reader thread is not a DPDK PMD spin-poller; it blocks between batches. Profiling shows the resulting scheduler events on SCC DPDK cores are swapper idle handoffs, not active preemption. This is benign. On hosts with ≥ 18 logical CPUs, allocate a dedicated overflow core to the SCC.

Disaster Recovery & Troubleshooting

Node Quarantine

If an SN begins throwing Bad Batch Notice (BBN) Control Messages, quarantine the node via the WebUX. The SCC will dynamically re-route queries to healthy replicas within the Storage Cluster.

Re-establishing Data Plane Sync

If an SN falls behind due to network partition on the NORM interface (UDP 6988), reboot the SN. The SCC will trigger a cold-boot sync event over standard TCP fallback on the Control Plane to ensure data integrity before marking the node Ready.

Download system_operators_manual.mdView raw