HD Administrator and Deployment Manual

Installing an instance? install/admin-guide.md is the guide to that: requirements, sizing, the cluster file, all three install paths and every option they take. This manual is the platform reference behind it — architecture, networking and storage configuration, the PKI model, and operational tasks.

System Overview & Architecture

HD — the Hyperscale Dataset Store — is developed and sold by FermiHDI Limited. It is an optimised storage engine for the records of a hyperscale dataset: one measured in billions of records and, at rest and unreduced, in terabytes to yottabytes. This manual covers deploying and operating an HD instance.

HD is an ultra-high-performance Data Management System designed for massive-scale analytics. It achieves extreme throughput by blending bare-metal acceleration technologies like DPDK (networking) and SPDK (storage) with a secure, Go-based control plane. This architecture bypasses traditional kernel bottlenecks, enabling a zero-copy data path where information flows from the network interface directly to storage or CPU cache layers.

Architectural Principles

The system is segmented into three distinct, isolated network planes:

1. Control Plane (Port 6986): An HTTPS mTLS network for all orchestration traffic. This includes node registration, configuration distribution, policy updates, and telemetry management. User data records never traverse this plane.
2. Data Plane (Port 6987): A high-speed, unidirectional network for binary data streaming using the proprietary FermiHDI HD Wire Protocol (FHDWP). Data flows from Storage Nodes directly back to the requesting client.
3. Internal DMS/CA Plane (Ports 6985, 6989): Secure channels used exclusively by the Data Management System (DMS) components to bootstrap nodes and provision cryptographic identities.

Core Components

The platform is composed of several specialized microservices:

• Data Management System (DMS): The central source of truth and orchestration.
  • DMS-Core: The Go-based engine that manages node topology, distributes schemas, and handles licensing.
  • DMS-CA: An integrated Certificate Authority that provisions short-lived mTLS certificates to all nodes, forming the cluster's root of trust.
  • DMS-WebUX: A Vue.js/Astro web interface (Port 3000) for system administration, configuration, and monitoring.
  • OTEL Collector: Receives every node's metrics, logs and traces over OTLP and re-exports them wherever an administrator configures — push (OTLP, syslog, Prometheus remote write, Kafka) or, for Prometheus, pull from a fixed port. See the observability page for what each node type actually emits.
• HD Proxy: The ingress gateway for all data queries. It enforces Policy-Based Access Control (PBAC) using an embedded Open Policy Agent (OPA), rewrites queries based on security rules, and forwards authorized requests to the storage layer.
• Storage Cluster Controller (SCC): A C++ load balancer and orchestrator for a group of Storage Nodes. It receives queries from the Proxy and ingest data from the Message Bus Ingester, distributing the workload across its assigned nodes. It uses NORM (NACK-Oriented Reliable Multicast) over UDP port 6988 for internal data distribution.
• Storage Node (SN): The C++-based data persistence and retrieval engine. It uses DPDK and SPDK to interact directly with networking and NVMe hardware, bypassing the kernel. It can offload filtering to GPUs (via SYCL) and hashing to specialized hardware like NVIDIA Bluefield DPUs.
• BloomRF Filter Node (optional): An engine role, at most one per storage cluster, that takes the SCC's ingest broadcast like a Storage Node and keeps a filter per batch instead of the records. At query time the SCC asks it which batches cannot match and leaves those unread. It persists nothing and is never allowed a false negative; see the filter node page.
• Message Bus Ingester: A high-performance C++ service that connects to Kafka, RabbitMQ or MQTT, or reads files from a directory or an S3-compatible store (JSON, CSV, Parquet, ORC, Avro, Protocol Buffers, XML), maps records onto the dataset schema and streams them to an SCC in batches; see the ingestion page.
• Query services: Front doors onto the Proxy — the SQL server (hd_sql: PostgreSQL wire, Arrow Flight SQL, HTTP), the GraphQL services (hd_graphql, hd_ipflow_graphql), the MCP server for AI assistants (hd_mcp) and the Data Explore web front end. Each carries the SDK, receives results straight from the Storage Nodes, and converts addresses at its own boundary; see the querying page.
• HD SDK: Client-side libraries (C++, Python, Go, JavaScript, Rust and others) that manage query execution. The SDK performs heavy computational work (SQL, Vector Search, Graph Math) locally on the data returned from the Storage Nodes, preventing computational bottlenecks within the storage cluster, and hands the developer the Arrow table itself.

System Topology Diagram

graph TD
    %% External Clients
    SDK_App[HD SDK Client]
    Client_Admin["DMS Admin UX (Port 3000)"]

    %% DMS Control Plane
    subgraph Control_Plane ["Management Layer (hd_dms)"]
        DMS((Data Management System))
        OpAMP_Server[OpAMP WebSocket Supervisor]
        DMS_CA[Internal Certificate Authority]
        DMS --> OpAMP_Server
        DMS --> DMS_CA
    end

    %% Network Entry and Routing
    subgraph Edge ["Network Edge (hd_proxy)"]
        Proxy[HD Proxy]
        OPA{Open Policy Agent}
        Proxy <--> OPA
    end

    %% Data Ingestion
    subgraph Ingestion ["Message Bus (hd_message_bus_ingester)"]
        Ingester[Message Bus Ingester]
        Kafka[(Kafka)]
        RabbitMQ[(RabbitMQ)]
        S3[(S3 / Filesystem)]
        Kafka -- "Zero-Allocation Parse" --> Ingester
        RabbitMQ -- "Reads Data" --> Ingester
        S3 -- "Reads Data" --> Ingester
    end

    %% Storage Cluster Components
    subgraph Storage_Cluster ["Storage Cluster (SC)"]
        SCC[Storage Cluster Controller]
        SN1[(Storage Node 1)]
        SN2[(Storage Node 2)]
        
        SCC -- "Load Balances Queries" --> SN1
        SCC -- "Load Balances Queries" --> SN2
    end

    %% Hardware Accelerators
    subgraph Acceleration ["Bare Metal / Acceleration"]
        DPDK[DPDK Networking]
        SPDK[SPDK Storage / OCF]
        SYCL[GPU SYCL Bitmask Filter]
        
        SN1 -. "Direct PCIe" .-> SPDK
        SN1 -. "Kernel Bypass" .-> DPDK
        SN1 -. "Compute Offload" .-> SYCL
    end

    %% Operations / Observability
    subgraph Telemetry ["Local Observability Stack Sidecars"]
        OTel(Local OTel Collector)
        CrashVault(Crash vault)
    end
    
    Central_Telemetry([FermiHDI gateway, Singapore])

    %% Data Flow Connections
    SDK_App -. "REST API / Custom TCP" .-> Proxy
    Client_Admin -. "HTTPS Config / APIs" .-> DMS
    
    Proxy == "Authorized / Modified Query" ==> SCC
    Ingester == "FHDWP Form 1 — TCP, 1200-byte MTU (Jumbo: 8500)" ==> SCC
    
    %% Control Flow Connections
    DMS_CA -- "mTLS Key Provisioning" --> Proxy
    DMS_CA -- "mTLS Key Provisioning" --> Ingester
    DMS_CA -- "mTLS Key Provisioning" --> SCC
    DMS_CA -- "mTLS Key Provisioning" --> SN1
    
    %% Observability Paths
    Proxy -. "Metrics/Logs" .-> OTel
    Ingester -. "Metrics/Logs" .-> OTel
    SCC -. "Metrics/Logs" .-> OTel
    SN1 -. "Crash reports" .-> CrashVault
    OpAMP_Server -. "Dynamic Rules via W/S" .-> OTel
    
    OTel -. "mTLS Batches" .-> Central_Telemetry
    CrashVault -. "Only what an operator shares" .-> Central_Telemetry

Deploying an Instance

There are four supported ways to stand up an instance, and all four are driven from one file — install/cluster.yaml — which describes the whole deployment. install/admin-guide.md is the guide to all of them; what follows is the summary.

install/fermihdi-discover                        # what this host has
install/fermihdi-configure                       # what it should run
install/validate-cluster cluster.yaml            # is that coherent
sudo install/fermihdi-prepare cluster.yaml --apply   # make it so

cd install/ansible && ansible-playbook site.yml  # the DMS, then the nodes

What a node is actually given

Deploying by hand is a matter of getting one contract right, so it is worth stating plainly: a node holds no configuration of its own. What it is given is an identity, and everything else — record size, which SCC to talk to, which devices to use, the hash election — comes back from the DMS in the registration response.

docker run -d --name sc-1-scc --network host \
  -e FERMIHDI_NODE_NAME=sc-1-scc \
  -e FERMIHDI_DMS_URL=https://dms-core:6986 \
  -e FERMIHDI_BOOTSTRAP_TOKEN="$SCC_TOKEN" \
  -e FERMIHDI_CA_CERT_PEM="$(cat dms-ca-bundle.pem)" \
  -e MASTER_CPUSET=2-7 \
  -e FERMIHDI_CONTROL_PLANE_SUBNET=10.147.0.0/24 \
  -e FERMIHDI_DATA_PLANE_SUBNET=10.150.0.0/24 \
  -v /dev/hugepages:/dev/hugepages -v /dev/hugepages1G:/dev/hugepages1G \
  --ulimit memlock=-1:-1 --cpuset-cpus 0-1 \
  cr.fermihdi.io/hd/scc:2.0.3

Four things about that command generalise to every node:

• --cpuset-cpus is the overflow cores, not the poll cores. MASTER_CPUSET is what the process pins its reactors to, and those cores must stay out of the container's cgroup — the container runtime shim inherits it and will otherwise be scheduled onto them.
• Both hugepage mounts. /dev/hugepages is the 2 MB pool DPDK's EAL needs for networking; /dev/hugepages1G is the 1 GB pool SPDK uses for storage and its ring buffers. One mountpoint carries one page size, so they are separate.
• --ulimit memlock=-1:-1. The HD engine pins its arena; the default 64 KB stops it at startup. In Kubernetes this is the IPC_LOCK capability.
• The bootstrap token is per node, and the CA bundle is how the node knows it is talking to your DMS and not something else.

The Ansible layer builds exactly this command from cluster.yaml, which is the argument for using it rather than maintaining the above by hand.

Talos Linux

A machine running Talos has no shell, no SSH and no container runtime to run docker run against, so the host-preparation step cannot run on it. It is applied to it instead, over the Talos API, by a set of tools that read the same cluster.yaml and ask the same questions:

install/fermihdi-talos-discover -n 10.147.0.20 --format=tsv > facts.tsv
install/fermihdi-configure --facts facts.tsv --append cluster.yaml
install/validate-cluster cluster.yaml
install/fermihdi-talos-prepare cluster.yaml --apply
install/fermihdi-talos-k8s cluster.yaml -o values.yaml
helm install fermihdi install/helm/fermihdi-talos -f values.yaml

The interview and the checks are the bare-metal ones unchanged. What differs is that hugepages, driver binding and kernel modules become machine configuration, and that the kernel command line — isolcpus, the IOMMU, the 1 GB hugepage pool — is part of the boot image rather than a file, so applying it replaces the image and reboots the machine. install/talos.md is the whole of that path.

Sandbox Environment

For development and testing, a pre-configured sandbox environment is available. This automates the deployment of the entire stack.

cd ./tests/sandbox_dms
./start_sandbox.sh

To shut down and clean up the sandbox environment:

cd ./tests/sandbox_dms
docker compose down -v --remove-orphans

Bare-Metal Host Preparation for Accelerated Storage Nodes

For maximum performance, Storage Nodes (SN) should be deployed on bare-metal hosts with specific kernel and system configurations to support SPDK and DPDK. These steps must be performed on the host machine before launching the SN container.

IOMMU Configuration

The IOMMU (Input-Output Memory Management Unit) must be enabled in the system firmware (BIOS/UEFI) and via kernel parameters to allow direct device access (VFIO).

After editing GRUB configuration, update and reboot: sudo update-grub && sudo reboot.

Hugepages Allocation

DPDK and SPDK require pre-allocated hugepages for efficient memory management. Both 2MB and 1GB pages are necessary.

Add the following to your GRUB command line: default_hugepagesz=2M hugepagesz=2M hugepages=1024 hugepagesz=1G hugepages=4

This allocates 1024 pages of 2MB (2 GiB total) for DPDK and 4 pages of 1GB (4 GiB total) for large DMA buffers.

CPU Isolation

To prevent OS scheduler noise from interfering with the data plane, isolate the CPU cores that will be assigned to the Storage Node.

Add the following to your GRUB command line (example reserves CPUs 2-15): isolcpus=2-15 rcu_nocbs=2-15 nohz_full=2-15

Host Setup

install/fermihdi-prepare applies all of the above from cluster.yaml: the hugepage pools per NUMA node, the hugetlbfs mounts, the vfio-pci bindings and the GRUB command line. It is idempotent, it reports what it would change before changing anything, and it never reboots — --verify is the pass to run after one. That is the supported path, and the one the Ansible layer drives.

install/validate-cluster cluster.yaml
sudo install/fermihdi-prepare cluster.yaml            # dry run
sudo install/fermihdi-prepare cluster.yaml --apply
install/fermihdi-prepare cluster.yaml --verify        # after the reboot

scripts/host_setup.sh remains for working on a machine by hand — deciding what it can do, or fixing one thing at a time:

# Check the current system status
sudo bash scripts/host_setup.sh status

# Configure hugepages and add to /etc/fstab
sudo bash scripts/host_setup.sh hugepages

# Bind an NVMe device to the vfio-pci driver for SPDK
sudo bash scripts/host_setup.sh bind 0000:01:00.0

# Get a recommended GRUB command line for the host
sudo bash scripts/host_setup.sh grub-advice

Networking & Storage Configurations

Networking

Port Assignments

FHDWP MTU Configuration

The FermiHDI HD Wire Protocol (FHDWP) on the Data Plane (port 6987) has strict MTU requirements to prevent IP fragmentation.

Records are never split across frames. Standard mode packs records into 1133 bytes of payload per frame, Jumbo Frame mode into 8433. A record wider than the payload travels alone in a frame of its own; the data plane runs over TCP, so that frame is segmented to the network's MTU like any other. The widest record a dataset may declare is 8433 bytes, in either mode.

[!WARNING] The MTU setting is instance-wide. All nodes in a cluster must use the same MTU. Mixing modes will lead to packet loss and data corruption. Do not enable Jumbo Frame mode in cloud environments (AWS, GCP, Azure) as they do not provide reliable end-to-end support.

To enable Jumbo Frame mode, the entire data path must be verified first.

Pre-flight Checklist:

1. Verify host NIC MTU: ip link show <interface> (should show mtu 9000).
2. Verify path MTU between nodes: ping -M do -s 8472 <destination_ip>. This must succeed without fragmentation.
3. For Docker, verify network MTU: docker network inspect <network_name> | grep -i mtu.

Changing the MTU on a running instance requires a full cluster restart.

Storage Node Configuration

Storage Nodes are configured via a local config.json file.

Example config.json:

{
  "node_name": "us-east-sn-01",
  "dms_url": "https://dms-core:6986",
  "command_port": 6986,
  "production_port": 8000,
  "production_interface": "0000:04:00.0",
  "drive_strategy": "fill_sequential",
  "storage_devices": [
    { "name": "nvme_0", "address": "0000:01:00.0", "type": "nvme" }
  ],
  "use_ocf": true,
  "ocf_cache_devices": ["nvme_0"],
  "ocf_core_devices": ["hdd_bulk_1"],
  "filter_device_selector": "gpu",
  "hash_offload_platform": "nvidia_bluefield"
}

Key Parameters:

• dms_url: The address of the DMS for registration.
• production_interface: The VFIO PCIe address of the network card for DPDK.
• storage_devices: A list of storage devices, identified by their PCIe address.
• use_ocf: Set to true to enable the Open CAS Framework, a caching tier that places fast devices (ocf_cache_devices) in front of slower, larger ones (ocf_core_devices).
• filter_device_selector: Offloads query filtering to gpu, cpu, or host.
• hash_offload_platform: Specifies hardware for cryptographic hash offloading.

Security & TLS/PKI Requirements

mTLS and Certificate Authority

HD operates on a zero-trust model, with all inter-node communication secured by mutual TLS (mTLS).

• DMS-CA: The DMS includes an integrated Certificate Authority which acts as the root of trust for the entire cluster.
• Node Registration Lifecycle:
  1. A new node starts and connects to the DMS Control Plane (port 6986) using standard TLS.
  2. The node sends an authenticated Certificate Signing Request (CSR) to the DMS.
  3. The DMS-CA signs the CSR and returns a short-lived X.509 certificate to the node.
  4. The node uses this certificate to authenticate itself for all subsequent communication within the cluster.
• Criticality: The DMS-CA is critical for cluster operation. If the DMS-CA is lost, the entire cluster must be rebooted to regenerate the trust chain.

Policy-Based Access Control (PBAC) with OPA

The HD Proxy is the enforcement point for data governance, using Open Policy Agent (OPA).

• Policy Management: Policies are written in Rego and managed through the Security > Governance tab in the DMS WebUX.
• Dynamic Updates: When a policy is updated in the DMS, it is pushed to all active Proxies via a POST /v1/proxy/config webhook, which triggers an atomic in-memory swap of the ruleset.
• Query Enforcement:
  1. A client sends a query to the query front door — the customer's reverse proxy, Traefik in the reference deployment.
  2. The front door authenticates the caller and sets the identity headers (X-Forwarded-User, -Role, -Groups, -Email), stripping any the client sent; see the front doors page.
  3. The HD Proxy receives the request and evaluates it against the active OPA policy.
  4. Instead of rejecting queries, the Proxy rewrites them:
     • Restricted Fields: If a user is not allowed to see a field (e.g., credit_score), the proxy overwrites that part of the query payload with null bytes (\x00), making it cryptographically unmatchable.
     • Enforced Fields: The proxy applies hard constraints (\xff) to enforce filters (e.g., tenant_id = 'abc') without dropping the connection.

Troubleshooting & Operational Tasks

Troubleshooting

• Node Quarantine: If a Storage Node begins emitting Bad Batch Notice (BBN) control messages, it can be quarantined via the DMS WebUX. The SCC will automatically re-route queries to healthy replicas.

• Data Plane Sync Issues: If an SN falls out of sync on the NORM multicast network (UDP 6988), rebooting the node will trigger a cold-boot synchronization process over the Control Plane to ensure data integrity before it is marked as Ready.

• AVX-512 Illegal Instruction Crashes: On systems using Windows Hyper-V, including Docker Desktop on Windows, the hypervisor can report AVX-512 support that the guest does not actually have, and an engine role then exits with SIGILL at startup. This affects that one virtualisation environment; it does not occur on bare-metal Linux or on a hypervisor that reports CPU features correctly.

  Run engine roles on bare-metal Linux or a correctly configured hypervisor. Where the environment cannot be changed, contact FermiHDI support for images built for a host without AVX-512 — the published images are built for the full instruction set and the alternative is supplied rather than produced locally.

• Viewing Logs: To view the logs for a specific component in a Docker Compose environment:

  docker compose -f ./tests/sandbox_dms/docker-compose.yml logs -f dms-core
  

Operational Tasks

• Observability: All nodes expose Prometheus metrics and health endpoints on port 8080. The system is designed to integrate with a central OpenTelemetry (OTel) Collector for logs, traces, and metrics, and holds every crash report in a crash vault on the DMS host.
• SDK Resource Management: The HD SDK uses asynchronous C++ reactors. To prevent resource leaks and freezes, especially in garbage-collected languages like Python, developers must clean up connections properly. Use context managers (with client:) or explicitly call client.stop() in a finally block.
• License Activation: An Enterprise License must be imported via the DMS WebUX to activate full functionality, including node registration and telemetry.