Administrator and Deployment Manual

This manual provides system administrators with the information required to deploy, manage, and operate an HD instance. It covers the system architecture, installation procedures for bare metal and Kubernetes, and essential operational tasks.

For detailed hardware and software prerequisites, sizing information, and a complete reference for every field in the cluster.yaml file, see the requirements and cluster file guide.

System Overview & Architecture

HD is a record store designed for ingesting and querying enormous volumes of small, uniform records at high performance. This is achieved by bypassing the operating system kernel for critical data path operations: network packets are handled by DPDK and disk I/O by SPDK, which writes directly to NVMe queues. This design requires specific host configurations, such as dedicated CPU cores and pre-allocated hugepage memory, which are applied during the installation process.

The administrator's guide describes the system's components, its three communication planes, and the data flows for ingest and query.

For a complete architectural breakdown, including data flows for ingest and query, refer to the system architecture guide.

How an Instance is Deployed

The supported path for deploying an HD instance is the install/ tooling. The entire deployment is described by a single cluster.yaml file, which serves as the source of truth for all tools. The process involves four main steps to prepare the hosts, followed by a deployment layer that runs the containers.

The installer guide details the deployment pipeline, the tools for each step, and the Ansible and Helm layers used for deployment.

For a detailed explanation of what each tool does and where decisions are made, see the installer guide.

Talos Linux

A cluster whose machines run Talos Linux is described by the same cluster.yaml and configured through the same interview process. However, because a Talos machine has no shell or SSH, the tooling interacts with it over the Talos API. The equivalent tools are fermihdi-talos-discover, fermihdi-talos-prepare, and fermihdi-talos-k8s. The full process is detailed in the Talos installation guide.

Deployment Instructions

Bare Metal

The Ansible layer documentation provides a complete guide to deploying on bare metal, including building the cluster.yaml file, setting secrets, and running the playbooks. For a complete guide to the playbooks, inventory, and required secrets, see the Ansible layer documentation.

Kubernetes

The Helm chart documentation explains how to generate a Helm values file from cluster.yaml and deploy the instance on Kubernetes. For details on what differs and how the chart is structured, see the Helm chart documentation.

Networking & Storage

Networking

HD operates on three network planes, which are defined by CIDRs in the networks block of cluster.yaml. Nodes automatically discover their own IP addresses on each plane by matching their network interfaces against these CIDRs.

Key ports used by the system are detailed in the administrator's guide. The most critical are:

Clients must be able to reach Storage Nodes directly on the data plane, as query results are streamed from the SNs to the client, not relayed through the Proxy.

Storage

For detailed storage and networking requirements, including sizing, see the administrator's guide.

Security & TLS/PKI

All communication between HD components is secured with mutual TLS (mTLS). The entire instance operates under a single certificate chain, where the deployment-specific DMS-CA signs certificates for every node.

• Identity: Nodes prove their identity to the DMS on first contact using a one-time bootstrap token. Once admitted, they receive a signed certificate which they use for all subsequent communication.
• Secrets: The Ansible layer guide explains how secrets are managed using environment variables, which are referenced in cluster.yaml but never stored there, and lists the variables required for a full deployment.
• User Authentication: HD does not authenticate end-users directly. Instead, it relies on two external reverse proxies, or front doors, which are responsible for authenticating users and setting trusted X-Forwarded-* identity headers on requests to the WebUX and the HD Proxy. For the full contract these front doors must fulfill, see the front doors guide.

Troubleshooting & Operational Tasks

Verification

The Ansible layer documentation explains how to use the verify.yml playbook to check host configurations across the cluster.

Common Tasks

The administrator's guide provides instructions for common operational tasks such as upgrading an instance, adding a node, and performing backups.

• License Management: New licenses are applied through the DMS WebUX. See the WebUX guide for this and other administrative tasks.

Common Issues

The administrator's guide includes a table of common issues with their likely causes and solutions.

The Talos installation guide provides instructions for debugging a Talos cluster using API-driven tools.