The DMS WebUX¶
The WebUX is the administrator's front end to the Data Management System (DMS): the one place an HD instance is defined, licensed, given its dataset schema and its nodes, and watched. Every page is a view onto DMS-Core's management API, so anything done here can also be done by a script against that API — and nothing done here reaches a node until DMS-Core pushes it, which the pages say when they do it.
This is a walk-through in the order a new instance is brought up, followed by the pages an operator returns to. It assumes the DMS plane is running and the WebUX is reachable through its front door; how the plane is deployed and who may reach it are the installer's subject.
Signing in and finding your way¶
The WebUX holds no accounts and applies no authorization of its own: whoever
reaches it is an administrator. What stands between it and the network is the
DMS front door — a reverse proxy with an authentication step that
terminates TLS, authenticates the administrator against your identity
provider, and sets X-Forwarded-User, X-Forwarded-Email and
X-Forwarded-Role on the requests it forwards, stripping any the browser
sent. The reference deployment and the lab use Traefik with a forward-auth
service for this; it is yours to provide and may be any ingress that keeps
that contract, with or without a forward-auth mechanism. It is separate from
the query front door in front of the Proxy, and it is never given a
certificate by the DMS-CA. The lab's authenticator answers "admin" for
everyone and must not be copied into a deployment. The navigation bar carries
seven pages:
| Page | Path | What it is for |
|---|---|---|
| Dashboard | / |
one instance at a time: health, utilisation, query history, the topology |
| Deployments | /deployments |
the instances themselves: create, dataset schema, the query filter mode |
| Nodes | /nodes |
every node of an instance: provision, approve, configure, revoke |
| Rego Policies | /policies |
the access-control policy the Proxy enforces |
| Telemetry & Logs | /telemetry |
opt-in sharing, DMS-Core's live log, exception tracking |
| OTEL Collector | /otel-collector |
where the instance's traces and metrics are exported |
| License | /license |
activation, and what the licence allows |
The Nodes page is reached from the navigation bar's Dashboard and Deployments pages ("View Nodes") as well as directly.
First: the licence¶
DMS-Core admits no node until a licence is installed, so the License page is where a new instance starts. A licence is issued to your organisation, is encrypted to this DMS's identity, and says how many nodes of each type it may admit.
Online activation. Enter the client id and licence id you were given and activate. The DMS fetches the licence, verifies it, and installs it.
Offline activation, for an instance with no route out. Download Request produces a request file carrying this DMS's identity; send it to FermiHDI, and Upload License installs the licence that comes back.
Either way, applying a licence through this page does more than store it: DMS-Core extracts the DMS-CA's intermediate certificate and its own identity certificate from the licence, which is the Day-0 bootstrap. A licence supplied to DMS-Core through its environment is verified and stored and nothing else.
Once installed, the page shows the licence's client, validity and utilisation: for each node type, how many are registered against how many the licence allows, with a type at its limit highlighted. A node type with no limit shows as unlimited.
Renew or Replace License runs the same activation again for a renewal, a change of term, or a change of the authorised node types and counts. It is issued against the DMS-CA's existing key, so every certificate already signed stays valid; applying it installs the renewed ICA on the DMS-CA, re-enrols the DMS's own services, and carries the new ICA to every node in its next configuration push. New counts apply to the next provisioning; running nodes are never evicted, and a type at or over its new limit shows red in the utilisation panel.
The types are the ones the nodes register as: HD_SCC, HD_SN, HD_INGEST
and HD_PROXY, plus HD_CLIENT for an application built on the SDK that
registers directly rather than querying through the Proxy (a custom ingester
is one), and the HD_ANY pool a type draws on when it has no element of its
own. HD_BLOOMRF is the exception: it is taken not by the filter node, which
is always authorised, but by each SCC that puts its filter service into
active mode, and given back when the SCC leaves it. An SCC refused the slot
runs the filter passive and says so in its log.
Second: the deployment¶
A deployment is one HD instance: a name, an instance id, and everything the nodes of that instance share. On the Deployments page, Register New Deployment asks for the three; the instance id is what every node is told to register into, so choose it before any node is configured.
Registration fixes two things that can never change afterwards: the hash election (the algorithm and chain block size every SCC, Storage Node and filter node computes over each batch). They lock the moment the first storage cluster is added.
The dataset schema¶
Each deployment card shows whether it has a dataset schema and, when it does,
the record length and the schema generation. A deployment with no schema is
not cosmetic: DMS-Core falls back to a test layout built into the image, and
the whole fleet runs on it. Dataset Schema → opens the editor: upload a file,
paste the JSON, or paste a schema generator share link -- the link the
generator's Share button gives carries the document in its #s= fragment,
which never reaches a server, and Import fills the box with it -- then
Upload Schema.
The schema decides how every node reads a record. Uploading one that keeps the record layout is a routine change: the nodes are updated in place and the card reports how many. Uploading one that changes the record length is not, and the page refuses it first, with the numbers: the current and requested record length, and how many SCC and Storage Nodes are running. Every record already stored was written at the old layout and becomes unreadable, so the change requires the acknowledgement offered below the warning, and then a rebuild of the data plane:
- SCC and Storage Nodes: reset their hashes and key-value stores and re-register; they draw new licences and certificates. Storage Nodes also reset their write pointers so old records are overwritten.
- Ingesters, Proxies and SDK clients: restart only.
- A filter node: restart; its store rebuilds from the stream.
The deployment's node registrations are forgotten and its licence utilisation resets to zero, so the rebuilt nodes are not refused as over capacity. The licence, the DMS-CA's certificate and the deployment itself are kept. The schema generation increments, which is how a node that was down during the change learns of it when it returns.
The query filter mode¶
Each deployment card carries the Query filter (BloomRF) setting: how far
every SCC of the instance trusts the cluster's filter node. It is set per
instance rather than per SCC because passive and active are only meaningful
when every controller agrees; the Proxy reports disagreeing controllers as
mixed.
| Setting | The SCC |
|---|---|
| Default | active wherever a filter node is provisioned; an SCC with no filter node is off whatever it is told |
| Active | consults the filter node and leaves the excluded batches unread |
| Passive | consults it, records what it would have excluded, and reads every batch anyway — for measuring a new filter node against real queries |
| Off | never consults it |
Changing the setting pushes the new mode to every SCC of the instance at once and the card says how many were reached. An SCC learns of the filter node itself only when its data plane starts, so the mode takes effect on an SCC that already knows its filter node, and a cluster whose filter node was added after the SCC started needs the SCC restarted first — the Nodes page shows which is the case.
Third: the storage clusters¶
On the Nodes page, choose the instance in the selector and + New Cluster. A storage cluster is one SCC and the Storage Nodes it fans records out to, and optionally one filter node. The dialog asks for the cluster's name and which Ingest and Proxy nodes should route to it; the hash algorithm shown is the deployment's election. Creating the first cluster locks that election.
Name clusters plainly — sc-1, sc-2 — because node names are derived from
them: sc-1-scc, sc-1-sn-1, sc-1-bloomrf. The Nodes page groups nodes
under their cluster, and the Dashboard's topology diagram derives the cluster
from the SCC's name.
Fourth: the nodes¶
Provision New Node declares a node before it exists, so that when the process starts and registers, DMS-Core knows what it is, which instance and cluster it belongs to, and how to trust it. The dialog asks for:
- Node name, which the process must present as its
FERMIHDI_NODE_NAME. - Node type. Storage Node, Storage Cluster Controller, BloomRF Filter Node, Ingest Node, Proxy Node.
- HD instance, and for an SCC, a Storage Node or a filter node, the storage cluster it joins.
- For a Storage Node, optionally an existing Storage Node to copy the configuration from: the storage strategy, the devices and the alert thresholds.
- The registration security mode, which decides how the node proves itself when it first connects:
| Mode | Then |
|---|---|
| Manual approval | the node registers and waits in Awaiting Approval until an administrator approves it here |
| Bootstrap token | the dialog shows a one-time token to give the node as FERMIHDI_BOOTSTRAP_TOKEN; it is admitted the moment it presents the token |
| Pre-generate credentials | the dialog produces the node's key and certificate now, for an air-gapped host where the process cannot register itself |
| K8s identity | the node presents its service-account token and is admitted on that |
A node is admitted only if the licence has room for its type; otherwise the dialog says which limit was reached.
The filter node¶
A BloomRF filter node is provisioned exactly like a Storage Node: type BloomRF Filter Node, the instance, and the cluster whose SCC it will serve. One per cluster, named after it. The dialog explains the one rule that matters: the node must be ready before the SCC's data plane starts, because the SCC takes its relay list once. DMS-Core holds that list until every provisioned Storage Node and filter node of the cluster reports ready — so a provisioned filter node that never starts holds the whole cluster's ingest with it. Provision one only when it is about to run; to add one to a running cluster, provision it, start it, wait for FullyReady, then restart the SCC.
Once it runs, the SCC's detail pane shows the Query Filter card: the mode
in force, the filter node the SCC was given, and its lookup budget. off (no
filter node) on a cluster that has one means the SCC started first.
Approving, watching and changing a node¶
The topology pane groups nodes: anything Awaiting Approval first, then Ingest nodes, Proxy nodes, each storage cluster with its SCC on top, and the DMS's own services in their panel. A banner counts the approvals waiting. Selecting a node shows its status, type, host, last heartbeat, when its certificate expires, traffic, resource utilisation and — for a Storage Node — its devices. Node certificates are signed for 30 days and every node renews its own within the last ten days, at startup and once a day, taking the new one in place without a restart; the expiry turns amber inside that window and red once passed, when the node has to register again. The pane offers:
- Approve Registration, for a node in manual-approval mode.
- Configure, which opens the node's settings. What is offered depends on the type: a Storage Node's storage strategy, devices and write path; an SCC's ports, interfaces, record size, batch timeout, fan-out pipeline, cores and filter lookup budget; an Ingest node's source and targets; a Proxy's routing and policy settings; a filter node's ports and log level. A setting the node's own environment has locked is marked and cannot be changed here. Saving pushes the configuration to the node.
- Graceful Shutdown, which asks the node to drain and stop.
- Revoke, which withdraws the node's certificate. A revoked node cannot reconnect until it is provisioned again.
A whole cluster can be taken Offline (maintenance) and brought back with Resume from its group header. Ingest and Proxy nodes are told at once and route around it; the SCC, its Storage Nodes and its filter node are marked accordingly.
The Dashboard¶
The Dashboard shows one instance at a time, chosen in the selector at the top. The summary row is the instance's active and healthy node counts, storage used, average CPU and total queries served. Below it: the instance's details, utilisation, a chart of query history, and the topology — Ingest nodes feeding storage clusters feeding Proxies. Clicking a storage cluster in the diagram drills into it: the SCC, the Storage Nodes it syncs, and the filter node when the cluster has one. Refresh re-reads everything.
Rego Policies¶
The Proxy evaluates every query against a policy written in Rego, compiled and served by the DMS's OPA engine. The page is an editor and a compiler: write or paste the policy, and Deploy compiles it and hands the bundle to the engine, from which every Proxy fetches it. A policy that does not compile is refused with the compiler's message and nothing changes.
Telemetry & Logs¶
Cloud sharing is the opt-in for sending anonymised metrics and exceptions to FermiHDI. It is off unless turned on here, and the decision is held by DMS-Core, not by the browser: the switch shows what DMS-Core holds, and every console that opens this page sees the same answer. The page also shows whether the deployment is registered with FermiHDI, the region it shares into, and the deployment identity it shares under.
Turning the switch on is refused while the DMS was installed with telemetry
sharing off (the installer's fermihdi_dms_enable_analysis setting, or
DMS_ENABLE_ANALYSIS on DMS-Core); the page says so and names the setting.
What the switch changes is one thing: what the deployment's OpenTelemetry collector forwards. With it on, DMS-Core has the collector forward the anonymised, shareable telemetry to FermiHDI once the deployment is registered; with it off, the collector forwards nothing. No node or service is told, and none changes what it reports to the deployment's own collector.
The switch also governs this console's own usage telemetry. Off, the WebUX
collects nothing. On -- or when the WebUX runs with the FERMIHDI_OPT_IN
environment flag -- the WebUX server reports its request timings and the
browser reports page loads and the console's own API calls to the
deployment's OpenTelemetry collector, the same one every node reports to, over
whichever protocol the collector is configured for. Nothing that identifies a
person is included: no addresses, no user agents, no identity headers, and
URLs reduced to their paths.
What is shared, and what never is¶
Three kinds of data, and each has its own rule.
| Tier | What it is | Where it goes | Consent |
|---|---|---|---|
| Never leaves | Schemas, dataset and node names, addresses, user identities, query text, record contents, and anything a console shows. | Nowhere. The collector removes it before anything is forwarded, and FermiHDI's gateway drops any key that is not on the published list below. | None needed: it is not collected. |
| Operational metadata | The licence ID, product and build commit, node and cluster counts, feature-usage counters, latency histograms, error types, and crash stacks without memory. | FermiHDI's own service in the region the deployment chose (Singapore), on hardware FermiHDI controls. | The switch on this page. |
| Crash reports | A crash's memory regions, which can hold record bytes, key material or a query in flight. | Held in this deployment's crash vault. Forwarded only per incident, only when an operator presses Share full report. | Per incident, every time. |
Only these attribute keys can leave with the operational metadata; anything
else is removed by the collector before it is sent: service.name,
service.version, service.namespace, deployment.environment,
fermihdi.build.commit, fermihdi.product, fermihdi.license_id,
fermihdi.region, host.arch, os.type, fermihdi.opt_in,
http.request.method, http.response.status_code, http.method,
http.status_code, http.route, rpc.method, rpc.service, error.type,
fermihdi.feature, fermihdi.node_role, log.level, event.category,
event.action, gen_ai.operation.name, gen_ai.tool.name.
FermiHDI runs its service on DigitalOcean in Singapore. DigitalOcean is the only infrastructure provider involved and has no access to the data at the product level; no other software-as-a-service provider processes any tier.
Crash reports¶
Every crash a service in this deployment reports lands in the deployment's crash vault, on the DMS host, and stays there. The Crash reports panel lists them: when, which service and version, what happened, the innermost frame, and what has been done with each. Open one to see its stack, symbolicated on the box against the symbol set shipped with the release, and a preview of exactly what each of two sizes would send:
- Stack only -- the symbolicated frames, the release and the tags. No memory, no attachments.
- Full report -- the report as received, memory regions included.
With cloud sharing on, the stack-only report is sent on its own as soon as a crash arrives, because the same crash on six deployments after a release is something FermiHDI should know without six clicks. The full report goes only when you press Share full report, with an optional support ticket to tag it with, and only after a confirmation. Export to file writes the whole crash as a tar for a support conversation on an air-gapped deployment. Every share and export is written to an audit log the deployment keeps, with who asked and when, and the panel shows it under the crash.
The vault keeps crashes for 30 days or 2 GB, whichever comes first; the oldest goes when either limit is reached. Delete one sooner from the panel.
The DMS-Core activity log streams DMS-Core's own log into the page as it happens: registrations, config pushes, licence decisions, the storage-cluster completeness gate. It can be paused, resumed and cleared. It is the first place to look when a node registers and nothing follows.
OTEL Collector¶
Every service in the instance sends traces and metrics to the DMS's OpenTelemetry collector; this page decides where the collector sends them on. Add an exporter of one of the supported kinds — OTLP over HTTP or gRPC, syslog, Prometheus remote write, Prometheus scrape, Kafka — with its endpoint and credentials, edit or remove exporters, and Deploy to apply the set. Until deployed, changes are marked as pending and the collector runs on the previous set.
Every exporter except one is a push: the collector forwards data out to a
destination you name. Prometheus scrape is the exception — it turns the
collector itself into something an external Prometheus (or any scraper) pulls
from, at <collector-host>:9464/metrics. That port is fixed rather than
chosen here, because — unlike a push destination, which is just a URL the
collector dials — a scrape target needs the collector's own inbound port
opened in its deployment manifest, which happens once at install time, not
from this page. Only one Prometheus scrape exporter can be enabled at a time;
a second is rejected before it can reach the collector, since both would try
to bind the same port.
For what each node type actually sends the collector — the real metric
names, log fields and trace spans, not just the pipeline shape — see
docs/observability.md.
A syslog exporter sends RFC 5424 messages over TCP or UDP. Its Transport setting chooses syslog over TLS or plain syslog; left at By port, it uses TLS on port 6514, the syslog-over-TLS port, and plain syslog on any other port. The Facility is applied to every message together with each record's severity.
Where this leaves the nodes¶
Nothing on these pages configures a node's host. What a node's environment carries is identity — its name, where the DMS is, the token or certificate that proves it — and the DMS supplies the rest at registration and on every push. When something on a node looks wrong, the order to look is: the node's status and last heartbeat on the Nodes page, then the activity log on the Telemetry page, then the node's own log.