Installing HD on Talos Linux

Talos has no shell, no SSH, no package manager and no /etc. A machine is configured by sending it a document over an API and rebooted by asking it to reboot. Nothing in README.md's pipeline survives that unchanged, and yet almost all of it survives in shape: the same cluster.yaml, the same fermihdi-configure interview, the same validate-cluster, the same sizing model. What changes is the last mile.

This file is the map of that last mile. ADMIN_GUIDE.md is still the long form for everything that is not Talos-specific.

The pipeline

fermihdi-talos-discover → fermihdi-configure → validate-cluster → fermihdi-talos-prepare → fermihdi-talos-k8s → helm
   what the machine has     what it should run     is that            machine config,          values for the       deploy
   (over the API)                                  coherent?          schematic, upgrade       Talos chart

Two of those five are the bare-metal tools unchanged. fermihdi-configure and validate-cluster never touched a host in the first place — they read facts and write a file — so they work here as they are.

cluster.talos.example.yaml is a worked four-machine example. It differs from cluster.example.yaml in exactly two places: a top-level talos: block, and a talos: block per host.

Start to finish

# 0. Boot the machines from a Talos ISO. They come up in maintenance mode with
#    an API and no configuration. Nothing else is needed on them.

# 1. Look at one. Read-only. --insecure because it has no certificates yet.
install/fermihdi-talos-discover -n 10.147.0.20 --insecure

# 2. Once it is configured, look again — this pass has the topology.
install/fermihdi-talos-discover -n 10.147.0.20 --format=tsv > facts.tsv
install/fermihdi-configure --facts facts.tsv --append cluster.yaml

# 3. Check the answer.
install/validate-cluster cluster.yaml

# 4. See what would happen. Changes nothing, uploads nothing.
install/fermihdi-talos-prepare cluster.yaml
install/fermihdi-talos-prepare cluster.yaml --render-only   # offline, files only

# 5. Make it so. This reboots machines, one at a time.
install/fermihdi-talos-prepare cluster.yaml --apply

# 6. Confirm the machines came up as asked.
install/fermihdi-talos-prepare cluster.yaml --verify

# 7. Deploy.
install/fermihdi-talos-k8s cluster.yaml -o values.yaml
helm install fermihdi install/helm/fermihdi-talos -f values.yaml \
     --kubeconfig talos/kubeconfig

Step 1 and step 2 are the same command twice, and that is not an accident — see below.

Why a Talos install is two passes

Talos answers two different sets of questions depending on whether the machine has been configured:

Everything the sizing model needs — per-NUMA-node cpulists, per-node hugepage counts, SMT sibling pairs, IOMMU groups — lives in /sys and /proc, and none of it is readable until the machine has a configuration. What is readable is enough to choose an install disk and identify the devices, which is enough to write a first configuration.

So: inventory coarsely, configure, inventory properly, configure again. fermihdi-talos-discover says so rather than quietly emitting zeroes — a tsv produced in maintenance mode exits 2 and tells you not to feed it to fermihdi-configure.

What happens to host_prep

The host_prep block is unchanged from the bare-metal file. What changes is where each half of it ends up, and the line runs somewhere no other distribution puts one.

Three of the four files fermihdi-prepare installs to survive a reboot have no counterpart here, because they have no reason to exist. Talos reapplies machine.sysfs and machine.kernel.modules itself.

The kernel argument list is not written twice. fermihdi-talos-prepare imports grub_tokens() from fermihdi-prepare, so the same host_prep produces the same arguments whether they land in GRUB or in a UKI.

The trap: extraKernelArgs does nothing

This is the one thing worth knowing before anything else.

Since Talos 1.10, a UEFI machine boots systemd-boot and a Unified Kernel Image, and the kernel command line is inside the UKI. From the Talos boot loader documentation:

With systemd-boot, the .machine.install.extraKernelArgs field in the machine configuration is ignored, as kernel arguments are embedded in the UKI and cannot be modified without upgrading the UKI.

So machine.install.extraKernelArgs is not the answer, and neither is any other config field. isolcpus, intel_iommu=on and the 1 GB hugepage pool go into an Image Factory schematic, which is uploaded, returns a content-addressed ID, and names a custom installer image:

# what fermihdi-talos-prepare writes as talos-sc-1.schematic.yaml
customization:
  extraKernelArgs:
    - intel_iommu=on
    - iommu=pt
    - default_hugepagesz=1G
    - hugepagesz=1G
    - hugepages=28
    - isolcpus=2-29
    - rcu_nocbs=2-29
    - nohz_full=2-29
  systemExtensions:
    officialExtensions:
      - siderolabs/intel-ucode

curl -X POST --data-binary @schematic.yaml https://factory.talos.dev/schematics
# {"id":"a94d87ed…"}
talosctl upgrade --image factory.talos.dev/metal-installer/a94d87ed…:v1.11.2

The schematic is the new GRUB, and it is per host, because isolcpus and the 1 GB pool differ per host. Talos 1.13 adds grubUseUKICmdline so that legacy-BIOS machines converge on the same mechanism.

Two consequences worth planning around:

• Changing isolcpus on a running machine is an upgrade and a reboot. On bare metal it is a GRUB edit and a reboot; here the boot image itself is replaced. --apply does it one machine at a time, and never in parallel.
• A machine installed fresh needs no upgrade. The configuration it is given in maintenance mode already carries machine.install.image, so the installer that writes the disk is the schematic's own and the machine boots with the right command line the first time. fermihdi-talos-prepare detects this and says so rather than rebooting a machine that has just booted.

Air-gapped

talos.factory_url points at the Image Factory. Set it to a self-hosted one and nothing reaches the internet. Schematic IDs are cached under <out_dir>/schematics/ by content hash, so a re-run of a dry run needs no network at all, and --render-only contacts nothing except — if you have not said which — the machine itself, to ask whether it is Intel or AMD. Put intel_iommu=on or amd_iommu=on in host_prep.grub.extra_args and even that goes away.

VFIO takes three things, not one

On bare metal, binding a device to vfio-pci is one step and the container gets it through a bind mount. Here it is three, and missing any one of them fails differently:

1. The rebind. A PCIDriverRebindConfig document per PCI address. A udev rule with ATTR{driver_override} does not work: the kernel has already bound nvme or mlx5_core by the time udev sees the device. Talos does the unbind, the override and the rebind itself.

2. The modules. vfio_pci and vfio_iommu_type1, from machine.kernel.modules. The second one is the expensive lesson. SPDK's setup script tries to modprobe it from inside the container and cannot:

   failed to bind NVMe disk 0000:03:00.0: … modprobe: ERROR: could not insert 'vfio_iommu_type1': Operation not permitted

   and what follows is EAL: No supported IOMMU extensions found! EAL: VFIO support could not be initialized. A container has no business loading kernel modules; the machine configuration loads it and SPDK finds it already there. (Seen in the wild as longhorn#12709, which is the same SPDK-on-Talos case as ours.)

3. The device plugin. A rebind creates /dev/vfio/<group> on the host and nothing follows from that: a container does not get a host device by asking, the kubelet does not know it exists, and the scheduler has no reason to place the SN on the machine that has it. helm/fermihdi-talos runs a VFIO device plugin that advertises each pool as an extended resource, and the Pods request them by name.

fermihdi-talos-k8s builds the pools from the devices cluster.yaml already assigns, one per role and device kind (a bloomrf role holds a NIC at most, and only when it opts into DPDK):

sc:
  sn:
    vfioResources:
      fermihdi.io/sn_nvme: 2

The plugin selects on drivers: ["vfio-pci"] as well as the address list, which is not redundant: it is what stops a device whose rebind silently failed from being advertised, and a Pod being scheduled onto a machine where the NVMe is still on the kernel driver.

The easy path

An SN on kernel-path storage — SATA, SAS, FC, iSCSI, NVMe-oF — needs none of this. No rebind, no IOMMU, no device plugin, and no 1 GB pages either, because those devices reach the disk through bdev_aio and never DMA. It is the configuration that asks the least of Talos and the one to start with on a new machine. talos-sc-2 in the example is exactly that.

Pod Security Admission

Talos ships the API server enforcing the baseline profile with only kube-system exempt. baseline allows neither IPC_LOCK nor a privileged container, so the SCC and the SNs are rejected at admission — before scheduling, with a message about the namespace rather than about the Pod.

The chart labels the sc and edge namespaces privileged and leaves the DMS and extra namespaces on baseline, because nothing in those needs more. Set talos.privilegedNamespaces: false only if something else already labels them, or if the API server was given an admissionControl exemption instead.

The two hugepage pools

Which pool is mounted where is not a style choice. StorageEngine.cpp scans for 1 GB pages and, when it finds any, pins SPDK's EAL hugedir:

SPDK: detected N x 1GB hugepages — EAL hugedir pinned to /dev/hugepages (1GB pool). 2MB hugepages (networking pool) are excluded from SPDK.

So a component that uses 1 GB pages gets:

and a component with no 1 GB pages has one pool, on /dev/hugepages, which is where DPDK looks by default. These are the same two paths fermihdi-prepare creates on bare metal, for the same reason.

Getting it backwards does not fail loudly. SPDK quietly allocates its NVMe DMA buffers from the pool the network data path was sized for, and the symptom is a transmit stall under load a long way from the cause.

One arithmetic trap carries over from GRUB: hugepages=N on the kernel command line is a per-node figure that the kernel multiplies across NUMA nodes. A two-socket machine whose busy node needs 14 must reserve 28. validate-cluster enforces this.

CPU pinning

isolcpus takes cores off the scheduler. It does not take them out of the kubelet's allocatable set — without more, the kubelet will hand an isolated core to whatever asks first and the HD instance pinned there shares it.

fermihdi-talos-prepare writes the kubelet's static CPU manager into the machine configuration:

machine:
  kubelet:
    extraConfig:
      cpuManagerPolicy: static
      reservedSystemCPUs: 0-1          # the union of container_cpuset
      topologyManagerPolicy: single-numa-node
      topologyManagerScope: pod

reservedSystemCPUs is the union of the container_cpuset values cluster.yaml already declares — the housekeeping cores — and is not invented when they are absent: a wrong reservation is worse than none, so the tool warns and leaves the policy off instead. topologyManagerPolicy is only set when every instance on the host declares numa_node.

The Pods must then be Guaranteed to get whole cores, which means integer CPU requests equal to limits. The chart does that from the cpuset widths.

Changing cpuManagerPolicy on a running node needs the old state removed. The kubelet refuses to start when the policy in its config disagrees with /var/lib/kubelet/cpu_manager_state. This is a Kubernetes behaviour, not a Talos one, and nothing here deletes that file for you.

single-numa-node rejects rather than degrades. Once it is on, a Pod whose CPUs, hugepages and devices cannot all be satisfied from one NUMA node is refused with TopologyAffinityError and never runs. That is the intended behaviour — an SN reaching across the interconnect on every I/O is the thing numa_node exists to prevent — but it means a device assigned to the wrong node in cluster.yaml shows up as an unschedulable Pod rather than as slow throughput. validate-cluster warns about the mismatch first; this is what happens if the warning is ignored.

Verifying

--verify reads the running machines and compares them to the definition. It changes nothing:

install/fermihdi-talos-prepare cluster.yaml --verify

It checks four things, and each failure names its cause:

Run it before helm install. A Pod claiming a device pool with no members stays Pending with no other symptom, and the reason is upstream of Kubernetes entirely.

Support and debugging

There is no shell to gather evidence from, so gather it through the API:

install/fermihdi-talos-discover -n 10.147.0.20 --dump-dir support/     # every raw response
install/fermihdi-talos-discover --replay support/                      # read it back later
talosctl -n 10.147.0.20 support -O support.zip                         # Talos's own bundle
talosctl -n 10.147.0.20 dmesg
talosctl -n 10.147.0.20 get pcidriverrebindstatus

--replay reads a dump instead of a machine and probes nothing, which makes an inventory reproducible after the fact and reviewable by someone who cannot reach the cluster.

Known limits

• Neither the DPDK NIC path nor the device plugin has been exercised on real hardware from this tooling. The mechanisms are the documented ones and the generated configuration is checked against the schema and rendered end to end, but "renders correctly" is not "works": the accelerator device class, the PF-rather-than-VF NIC pool, and SPDK attaching through a plugin-provided /dev/vfio node are all worth confirming on a real machine before anyone depends on them.
• The front door is still not deployed — same gap as the general-purpose chart. See README.md.
• helm/fermihdi-talos is a fork of helm/fermihdi rather than a layer on top of it, so a fix to the DMS templates has to be made in both. That is deliberate for now: it keeps the working chart untouched while this one is new.
• ssh_user and ssh_key in cluster.yaml are ignored on Talos hosts. validate-cluster says so rather than letting them look meaningful.

Sources

• Boot loader — why extraKernelArgs is ignored under systemd-boot
• Boot assets and the Image Factory — schematics and metal-installer images
• Configuration patches — multi-document strategic merge, and the default PSA configuration
• Editing machine configuration — which fields apply without a reboot
• PCIDriverRebindConfig
• v1alpha1 machine config reference
• Talos for Linux admins — the talosctl equivalent of each familiar command
• The --insecure flag — what maintenance mode does and does not offer
• longhorn#12709 — SPDK, vfio_iommu_type1 and Talos