The DMS WebUX¶
The WebUX is the administrator's front end to the Data Management System (DMS): the one place an HD instance is defined, licensed, given its dataset schema and its nodes, and watched. Every page is a view onto DMS-Core's management API, so anything done here can also be done by a script against that API — and nothing done here reaches a node until DMS-Core pushes it, which the pages say when they do it.
This is a walk-through in the order a new instance is brought up, followed by the pages an operator returns to. It assumes the DMS plane is running and the WebUX is reachable through its front door; how the plane is deployed and who may reach it are the installer's subject.
Signing in and finding your way¶
The WebUX holds no accounts and applies no authorization of its own: whoever
reaches it is an administrator. What stands between it and the network is the
DMS front door — a reverse proxy with an authentication step that
terminates TLS, authenticates the administrator against your identity
provider, and sets X-Forwarded-User, X-Forwarded-Email and
X-Forwarded-Role on the requests it forwards, stripping any the browser
sent. The reference deployment and the lab use Traefik with a forward-auth
service for this; it is yours to provide and may be any ingress that keeps
that contract, with or without a forward-auth mechanism. It is separate from
the query front door in front of the Proxy, and it is never given a
certificate by the DMS-CA. The lab's authenticator answers "admin" for
everyone and must not be copied into a deployment. The navigation bar carries
seven pages:
| Page | Path | What it is for |
|---|---|---|
| Dashboard | / |
one instance at a time: health, utilisation, query history, the topology |
| Deployments | /deployments |
the instances themselves: create, dataset schema, the query filter mode |
| Nodes | /nodes |
every node of an instance: provision, approve, configure, revoke |
| Rego Policies | /policies |
the access-control policy the Proxy enforces |
| Telemetry & Logs | /telemetry |
opt-in sharing, DMS-Core's live log, exception tracking |
| OTEL Collector | /otel-collector |
where the instance's traces and metrics are exported |
| License | /license |
activation, and what the licence allows |
The Nodes page is reached from the navigation bar's Dashboard and Deployments pages ("View Nodes") as well as directly.
First: the licence¶
DMS-Core admits no node until a licence is installed, so the License page is where a new instance starts. A licence is issued to your organisation, is encrypted to this DMS's identity, and says how many nodes of each type it may admit.
Online activation. Enter the client id and licence id you were given and activate. The DMS fetches the licence, verifies it, and installs it.
Offline activation, for an instance with no route out. Download Request produces a request file carrying this DMS's identity; send it to FermiHDI, and Upload License installs the licence that comes back.
Either way, applying a licence through this page does more than store it: DMS-Core extracts the DMS-CA's intermediate certificate and its own identity certificate from the licence, which is the Day-0 bootstrap. A licence supplied to DMS-Core through its environment is verified and stored and nothing else.
Once installed, the page shows the licence's client, validity and utilisation: for each node type, how many are registered against how many the licence allows, with a type at its limit highlighted. A node type with no limit shows as unlimited.
Renew or Replace License runs the same activation again for a renewal, a change of term, or a change of the authorised node types and counts. It is issued against the DMS-CA's existing key, so every certificate already signed stays valid; applying it installs the renewed ICA on the DMS-CA, re-enrols the DMS's own services, and carries the new ICA to every node in its next configuration push. New counts apply to the next provisioning; running nodes are never evicted, and a type at or over its new limit shows red in the utilisation panel.
The types are the ones the nodes register as: HD_SCC, HD_SN, HD_INGEST,
HD_PROXY and HD_BLOOMRF, plus HD_CLIENT for an application built on the
SDK that registers directly rather than querying through the Proxy (a custom
ingester is one), and the HD_ANY pool a type draws on when it has no element
of its own.
Second: the deployment¶
A deployment is one HD instance: a name, an instance id, and everything the nodes of that instance share. On the Deployments page, Register New Deployment asks for the three; the instance id is what every node is told to register into, so choose it before any node is configured.
Registration fixes two things that can never change afterwards: the hash election (the algorithm and chain block size every SCC, Storage Node and filter node computes over each batch). They lock the moment the first storage cluster is added.
The dataset schema¶
Each deployment card shows whether it has a dataset schema and, when it does, the record length and the schema generation. A deployment with no schema is not cosmetic: DMS-Core falls back to a test layout built into the image, and the whole fleet runs on it. Dataset Schema → opens the editor: upload a file or paste the JSON, and Upload Schema.
The schema decides how every node reads a record. Uploading one that keeps the record layout is a routine change: the nodes are updated in place and the card reports how many. Uploading one that changes the record length is not, and the page refuses it first, with the numbers: the current and requested record length, and how many SCC and Storage Nodes are running. Every record already stored was written at the old layout and becomes unreadable, so the change requires the acknowledgement offered below the warning, and then a rebuild of the data plane:
- SCC and Storage Nodes: reset their hashes and key-value stores and re-register; they draw new licences and certificates. Storage Nodes also reset their write pointers so old records are overwritten.
- Ingesters, Proxies and SDK clients: restart only.
- A filter node: restart; its store rebuilds from the stream.
The deployment's node registrations are forgotten and its licence utilisation resets to zero, so the rebuilt nodes are not refused as over capacity. The licence, the DMS-CA's certificate and the deployment itself are kept. The schema generation increments, which is how a node that was down during the change learns of it when it returns.
The query filter mode¶
Each deployment card carries the Query filter (BloomRF) setting: how far
every SCC of the instance trusts the cluster's filter node. It is set per
instance rather than per SCC because passive and active are only meaningful
when every controller agrees; the Proxy reports disagreeing controllers as
mixed.
| Setting | The SCC |
|---|---|
| Default | active wherever a filter node is provisioned; an SCC with no filter node is off whatever it is told |
| Active | consults the filter node and leaves the excluded batches unread |
| Passive | consults it, records what it would have excluded, and reads every batch anyway — for measuring a new filter node against real queries |
| Off | never consults it |
Changing the setting pushes the new mode to every SCC of the instance at once and the card says how many were reached. An SCC learns of the filter node itself only when its data plane starts, so the mode takes effect on an SCC that already knows its filter node, and a cluster whose filter node was added after the SCC started needs the SCC restarted first — the Nodes page shows which is the case.
Third: the storage clusters¶
On the Nodes page, choose the instance in the selector and + New Cluster. A storage cluster is one SCC and the Storage Nodes it fans records out to, and optionally one filter node. The dialog asks for the cluster's name and which Ingest and Proxy nodes should route to it; the hash algorithm shown is the deployment's election. Creating the first cluster locks that election.
Name clusters plainly — sc-1, sc-2 — because node names are derived from
them: sc-1-scc, sc-1-sn-1, sc-1-bloomrf. The Nodes page groups nodes
under their cluster, and the Dashboard's topology diagram derives the cluster
from the SCC's name.
Fourth: the nodes¶
Provision New Node declares a node before it exists, so that when the process starts and registers, DMS-Core knows what it is, which instance and cluster it belongs to, and how to trust it. The dialog asks for:
- Node name, which the process must present as its
FERMIHDI_NODE_NAME. - Node type. Storage Node, Storage Cluster Controller, BloomRF Filter Node, Ingest Node, Proxy Node.
- HD instance, and for an SCC, a Storage Node or a filter node, the storage cluster it joins.
- For a Storage Node, optionally an existing Storage Node to copy the configuration from: the storage strategy, the devices and the alert thresholds.
- The registration security mode, which decides how the node proves itself when it first connects:
| Mode | Then |
|---|---|
| Manual approval | the node registers and waits in Awaiting Approval until an administrator approves it here |
| Bootstrap token | the dialog shows a one-time token to give the node as FERMIHDI_BOOTSTRAP_TOKEN; it is admitted the moment it presents the token |
| Pre-generate credentials | the dialog produces the node's key and certificate now, for an air-gapped host where the process cannot register itself |
| K8s identity | the node presents its service-account token and is admitted on that |
A node is admitted only if the licence has room for its type; otherwise the dialog says which limit was reached.
The filter node¶
A BloomRF filter node is provisioned exactly like a Storage Node: type BloomRF Filter Node, the instance, and the cluster whose SCC it will serve. One per cluster, named after it. The dialog explains the one rule that matters: the node must be ready before the SCC's data plane starts, because the SCC takes its relay list once. DMS-Core holds that list until every provisioned Storage Node and filter node of the cluster reports ready — so a provisioned filter node that never starts holds the whole cluster's ingest with it. Provision one only when it is about to run; to add one to a running cluster, provision it, start it, wait for FullyReady, then restart the SCC.
Once it runs, the SCC's detail pane shows the Query Filter card: the mode
in force, the filter node the SCC was given, and its lookup budget. off (no
filter node) on a cluster that has one means the SCC started first.
Approving, watching and changing a node¶
The topology pane groups nodes: anything Awaiting Approval first, then Ingest nodes, Proxy nodes, each storage cluster with its SCC on top, and the DMS's own services in their panel. A banner counts the approvals waiting. Selecting a node shows its status, type, host, last heartbeat, when its certificate expires, traffic, resource utilisation and — for a Storage Node — its devices. Node certificates are signed for 30 days and every node renews its own within the last ten days, at startup and once a day, taking the new one in place without a restart; the expiry turns amber inside that window and red once passed, when the node has to register again. The pane offers:
- Approve Registration, for a node in manual-approval mode.
- Configure, which opens the node's settings. What is offered depends on the type: a Storage Node's storage strategy, devices and write path; an SCC's ports, interfaces, record size, batch timeout, fan-out pipeline, cores and filter lookup budget; an Ingest node's source and targets; a Proxy's routing and policy settings; a filter node's ports and log level. A setting the node's own environment has locked is marked and cannot be changed here. Saving pushes the configuration to the node.
- Graceful Shutdown, which asks the node to drain and stop.
- Revoke, which withdraws the node's certificate. A revoked node cannot reconnect until it is provisioned again.
A whole cluster can be taken Offline (maintenance) and brought back with Resume from its group header. Ingest and Proxy nodes are told at once and route around it; the SCC, its Storage Nodes and its filter node are marked accordingly.
The Dashboard¶
The Dashboard shows one instance at a time, chosen in the selector at the top. The summary row is the instance's active and healthy node counts, storage used, average CPU and total queries served. Below it: the instance's details, utilisation, a chart of query history, and the topology — Ingest nodes feeding storage clusters feeding Proxies. Clicking a storage cluster in the diagram drills into it: the SCC, the Storage Nodes it syncs, and the filter node when the cluster has one. Refresh re-reads everything.
Rego Policies¶
The Proxy evaluates every query against a policy written in Rego, compiled and served by the DMS's OPA engine. The page is an editor and a compiler: write or paste the policy, and Deploy compiles it and hands the bundle to the engine, from which every Proxy fetches it. A policy that does not compile is refused with the compiler's message and nothing changes.
Telemetry & Logs¶
Cloud sharing is the opt-in for sending anonymised metrics and exceptions to FermiHDI; it is off unless turned on here, and the page shows whether the deployment is registered for it, what would be exported, and the deployment identity it would be exported under. With it on, View Setup Instructions produces the configuration for a local Sentry relay — as Docker Compose, Kubernetes manifests or a bare-metal unit — that forwards application exceptions.
The DMS-Core activity log streams DMS-Core's own log into the page as it happens: registrations, config pushes, licence decisions, the storage-cluster completeness gate. It can be paused, resumed and cleared. It is the first place to look when a node registers and nothing follows.
OTEL Collector¶
Every service in the instance sends traces and metrics to the DMS's OpenTelemetry collector; this page decides where the collector sends them on. Add an exporter of one of the supported kinds — OTLP over HTTP or gRPC, syslog, Prometheus remote write, Kafka — with its endpoint and credentials, edit or remove exporters, and Deploy to apply the set. Until deployed, changes are marked as pending and the collector runs on the previous set.
Where this leaves the nodes¶
Nothing on these pages configures a node's host. What a node's environment carries is identity — its name, where the DMS is, the token or certificate that proves it — and the DMS supplies the rest at registration and on every push. When something on a node looks wrong, the order to look is: the node's status and last heartbeat on the Nodes page, then the activity log on the Telemetry page, then the node's own log.