Observability: metrics, logs and traces

This page documents what each node type emits — Prometheus metrics, structured logs, and OpenTelemetry traces — and how the DMS's OTEL Collector routes them to a destination you choose. For configuring the collector's exporters, see the DMS WebUX guide's OTEL Collector section.

Every node in an HD instance does two independent things with its telemetry:

1. Serves its own /metrics and /healthz//livez//readyz directly, on its own observability port (8080 by default, OBS_PORT where the service honours the override — see the per-node notes below, since not every service does). A tool that scrapes nodes directly reads this.
2. Pushes metrics, logs and traces to the DMS's OTEL Collector over OTLP (gRPC 4317 or HTTP 4318, the OpenTelemetry standard ports). The collector then re-exports what an administrator has configured — see The OTEL Collector below, including its own new pull-based Prometheus endpoint.

These are two views of the same instruments on the C++ nodes (SCC, SN, BloomRF) — one render function serves both /metrics and the OTLP push, so they cannot disagree. On the Go services, the two paths are typically implemented separately (a promhttp handler for /metrics, a distinct OTLP trace exporter for traces), so the two can and sometimes do carry different information — noted per service below.

The OTEL Collector

The collector is one process per instance, run under DMS-Core's management (config pushed over OpAMP). It always receives; what it does with what it receives is entirely admin-configured.

Always on, every deployment:

Admin-configured exporters (DMS → Telemetry page, or the /v1/otel/config API), each a distinct destination:

With no exporters configured, the collector still receives everything (every node reports to it regardless) but forwards nothing — a debug sink at basic verbosity, so telemetry doesn't back up.

Prometheus (scrape) exporter

Every other exporter is a push: the collector dials out to a URL you supply. The scrape exporter is the one inbound exception, and it works differently because of that:

• Port is fixed at 9464, not admin-chosen. A pull target needs the collector's own port opened in its deployment manifest (the Helm chart's container/Service ports, or the bare-Docker installer's -p flags) — that happens once, at install time, the same way 4317/4318/13133 already are. DMS-Core cannot reopen a container port at runtime, so enabling the exporter on the Telemetry page is the only step left to you.
• At most one enabled at a time — a second would try to bind the same port and fail the collector's own startup. The config is rejected before it reaches the collector.
• Reachability depends on your deployment path: the Helm chart's Service exposes it cluster-wide (a ServiceMonitor from your own kube-prometheus-stack can scrape it with no further change); the bare-Docker installer publishes it to the host; the self-contained sandbox does not publish it to the host (it follows the sandbox's existing convention of publishing only the front doors and a couple of lab-only tools, nothing else).

Logging format

Every node in an HD instance — Message Bus Ingester, SCC, SN, BloomRF, the DMS suite, the Query Proxy, the SDK (and therefore any client built on it), hd_mcp, hd_graphql, hd_sql, and Data Explore — emits the same JSON log envelope. This is a real, enforced standard (.agents/rules/observability-standard.md), not just a shared convention that happened to converge: two independent components (the process writing a line, and the DMS Collector's exporter reading it back out of the ring on the C++ nodes) depend on it staying exact.

{
  "logging_format_version": "2.0.0",
  "timestamp": 1786656385,
  "ts_ms": 1786656385433,
  "ts_us": 1786656385433025,
  "level": "info",
  "service": { "name": "hd_storage_node", "version": "1.0.0", "environment": "production", "node_name": "sn-3" },
  "event": { "action": "sn_rx_first", "category": "ingest_ops", "outcome": "success" },
  "opt_in": false,
  "tracing": { "trace_id": "0af7651916cd43dd8448eb211c80319c", "span_id": "b7ad6b7169203331" },
  "message": "[INGEST] sn received first record of batch",
  "data": { "batch_id": "0000…6a7e4d9c", "records": "60000" }
}

Every field is always present except tracing, which appears only when the line has real trace context to attach — never fabricated locally, never emitted with empty strings. opt_in is likewise always present, always at the top level: never nested under a fermihdi key, never omitted when false.

Facts worth knowing before treating this as a wire contract:

• ts_us is the field to join on, not timestamp/ts_ms — an SN's own ingest window is single-digit milliseconds, which ts_ms doesn't resolve.
• message is never machine-parsed. It routinely quotes JSON fragments back at the reader; anything a consumer needs lives in data or event.
• data.batch_id is always the full 128 hex characters, never a truncated/local form — two different truncation conventions once made a cross-node join silently return zero rows instead of erroring.
• Only query-path records carry a tracing block; ingest records carry none — tracing is control-plane query-path only on the storage cluster, covered per node below.
• On SCC/SN/BloomRF/the Ingester, routing is: error-and-above plus startup lines go to stdout; everything else goes to the in-process ring buffer and out via the collector. FERMIHDI_LOG_STDOUT=all puts every level on stdout in addition to the normal export path.
• The exporter counts what it couldn't read: fermihdi.log.malformed (not valid JSON) and fermihdi.log.dropped (lost to ring wraparound before being drained), both labelled by the emitting role — a non-zero value here means an emitter and the schema have diverged.

How tracing gets populated differs by service, and that's expected. SCC/SN/BloomRF/the Ingester and the DMS suite (dms-core, dms-ca, dms-opa-engine) generate real trace context locally — the DMS suite by creating a matching span per log call (the log message becomes the span name, purely to attach trace_id/span_id; there's no separate tracing step), the storage cluster by propagating the query's root span. hd_proxy does the same from its own genuine per-operation spans (ServeHTTP, PBAC_Evaluation, SCC_Fanout — see its own section below). hd_sql, hd_graphql, hd_mcp, and the SDK carry no OpenTelemetry SDK at all, so none of them generate trace context — a tracing block only appears in their logs when a caller explicitly hands one in (e.g. from an inbound request header, or OTEL_TRACE_ID/OTEL_SPAN_ID for the SDK), never generated locally.

Whether a log line reaches the DMS Collector is a separate question from its shape, and doesn't change with this schema. Two groups:

• Always deployed, always pushed automatically: SCC, SN, the optional BloomRF filter node, the Message Bus Ingester, and the DMS suite push structured JSON logs to the DMS's OTEL Collector on their own (see The OTEL Collector above) — no configuration needed. This is the population the DMS Collector's log parsing actually depends on.
• Optional to deploy, local logs only unless you relay them yourself: hd_proxy, hd_sql, hd_graphql, hd_mcp, and any SDK-embedding client write this same JSON to their own stdout only — point your own log collection (a sidecar, a log-shipping agent, your container platform's stdout capture) at each one if you want them in your pipeline. hd_proxy is a partial exception: its logs stay local like the rest of this group, but its traces genuinely do reach the DMS Collector (see its own section below) — logs and traces are separate signals here. Data Explore is a further exception, and different from what you might expect: its server-side logs stay local like the rest of this group (and its server-side errors still go to Sentry only, unchanged), but its browser-side usage traces and structured logs reach the DMS Collector through a relay when the deployment has opted in, and its browser-side crash reports always reach the deployment's crash vault regardless of opt-in — see its own section below.

Metrics, logs and traces by node

Storage Cluster Controller (SCC)

Metrics (/metrics, port 8080, hardcoded — SCC's HTTP listener does not honour the OBS_PORT override):

queries_processed, records_scanned/records_selected and cpu_utilization/disk_utilization have no SCC-side equivalent — the SCC fans out and hashes; it does not itself scan or select records (that's the SN's job) and does not publish host-resource gauges.

A large additional set of dev-only metrics (queue depths, reactor busy/idle/steal time, NIC hardware counters, latency percentiles, memory-bandwidth/PMU samples) exists under FERMIHDI_DEV_METRICS/ FERMIHDI_DEV_METRICS_HOTPATH compile flags. Both default OFF in Release/distroless/rock builds — a production SCC image ships none of these; they exist for development-time bottleneck diagnosis only.

Logs: event.category values actually used include ingest_ops (scc_rx_first, scc_hash_closed, scc_rx_last) and query_ops (scc_q_trace, carrying a tracing block).

Traces: creates the query's root span (scc.query.dispatch) when a query is dispatched, with attributes for batch-ID count, QoS, and the SN sessions fanned out to. The trace ID is built from the issuing client's query_id. The ingest/data path creates no spans by design.

Health: /healthz, /livez, /readyz alongside /metrics, same port, served by a minimal raw-socket HTTP server in the OP (supervisor) process — the same process that also holds the metrics registry, so /metrics and the OTLP push read one shared set of instruments.

Storage Node (SN)

Metrics (/metrics, port 8080, hardcoded, same caveat as SCC):

Of the eight mandatory metrics in the observability rule, the SN is the one node that actually carries all of them (under the names above); cpu_utilization only exists as a dev-only PMU sample (FERMIHDI_DEV_METRICS), not in a production build. The same large dev-only metric set described for SCC exists here too (queue depths per pipeline stage, NIC counters, storage-device usage, latency percentiles, saturation counts) — same OFF-by-default gating.

Logs: ingest_ops (sn_validate_start/sn_validate_done, carrying data.batch_id) and query_ops (sn_q_trace, carrying a tracing block plus data.query_id, plus a family of sn_read-style actions on the read path).

Traces: continues the SCC's root span as a child span (sn.query.dispatch) via the inbound W3C traceparent, with query_id/batch-ID-count/QoS attributes. No spans on the ingest path, consistent with SCC.

Health: same shape as SCC — /healthz//livez//readyz//metrics on 8080 from the OP process; unhealthy means a child process died or missed a health-ping deadline (FERMIHDI_HEALTH_PONG_DEADLINE_MS, default 15s) — log volume/severity does not by itself affect health status.

BloomRF filter node

Metrics (/metrics, port 8080, hardcoded):

All of the above are declared # TYPE ... gauge in the exporter, even the ones that only ever increase (lookups, batches_closed, the verify_* and lookup_* counts). If you graph these, treat them as you would a counter (use them under rate()/increase()-style reasoning conceptually) rather than expecting a _total suffix or a PromQL counter type — the Prometheus type metadata itself says gauge.

BloomRF has no direct equivalent of queries_processed (it pre-filters, it doesn't execute queries), records_stored/records_scanned (it folds records into filters rather than storing or scanning them — closest analog is records_seen), or bytes_received/bytes_sent/cpu_utilization/ disk_utilization (none tracked; filter memory footprint is the closest via filters_bytes).

Logs: rich event set across startup (config received, filter server bound, filters reloaded from persistence, DMS registration), config (schema applied/changed — a schema-generation change discards existing filters), dataplane (channel state), ingest (filter_corrupt on a bad batch filter), verify (verification stream lifecycle and verdicts), and query_ops (bloomrf_filter, a per-RPC measurement event carrying candidate/evaluated/excluded counts and elapsed microseconds).

Traces: BloomRF sits in the query path between the SCC and its own FilterBatches RPC, and receives the SCC's trace context over gRPC metadata, but does not itself call StartChildSpan to attach a span object — it carries trace correlation only through the tracing block on its query_ops log lines, not as a span in the trace view itself.

Persistence: filter reload/discard state (filters_reloaded vs filters_not_persistent) is logged, not metriced — check /healthz or the startup log for filter-store health, not a Prometheus series.

Message Bus Ingester

Logs reach the DMS Collector automatically, the same as SCC/SN/BloomRF — it uses the same flat 2.0.0 shape and the same always-on logging path (see Logging format above).

Metrics: no /metrics endpoint by default. The observability server this service could run is disabled by default. Internally it does keep records_processed_total, packets_sent_total, and send_errors_total as in-memory counters, but these are only ever written into structured log lines (event.category: "application"), never exposed for scraping unless an operator changes that default.

Health: a --healthcheck CLI flag (not an always-running HTTP listener) connects to 127.0.0.1:<healthcheck_port> (default 8080) and expects a JSON {"status":"ok","timestamp":...} — this is what a container orchestrator's health probe should invoke, per the CLI-healthcheck convention documented in .agents/rules/observability-standard.md.

Traces: none. The ingester is a pure ingest-path component; per the control-plane-only tracing design, it creates and propagates no spans.

Logs: event.category: "application" throughout, actions log, telemetry, metric (a periodic uptime line, not a real metric), and batch (batch-ID rollover). config.opt_in (env FERMIHDI_OPT_IN or --opt-in) marks records for FermiHDI sharing the same way every other service's top-level opt_in field does.

DMS suite

The three main DMS Go binaries — dms-core, dms-ca, dms-opa-engine — share the same observability implementation, so they behave identically: /healthz, /livez, /readyz and /metrics on OBS_PORT (default 8080, and this one genuinely honours the env override). Two more DMS binaries do not:

• dms-opa-adapter installs only crash reporting — no health or metrics endpoint at all.
• dms-crash-vault is entirely outside this framework: plain-text /healthz (not JSON) on its intake port (3001) and management port (3002), and plain log.Printf instead of structured JSON logging.

Metrics: dms-core (and only dms-core) declares six Prometheus series matching the mandatory-metric names — queries_processed_total, records_stored_total, records_scanned_total, bytes_received_total, bytes_sent_total, memory_usage_bytes — but none are incremented or set; they exist on /metrics and read zero. DMS sits on the control plane, not the data path, so most of those names don't apply to it in the first place. For DMS-specific activity — node registrations, licence checks/issuance, OTEL config pushes, OpAMP connection count — there is no dedicated Prometheus series: go to the logs (below) for that activity, or the WebUX's Telemetry page, which reads the OpAMP supervisor's live connection count directly rather than through a Prometheus series.

Logs (logging_format_version: "2.0.0" — see Logging format above): representative event.category/event.action pairs actually logged include security/renew_cert, security/crl_check, security/revocation, lifecycle/node_decommission, lifecycle/node_provision, storage_cluster/capacity_degraded, topology/graceful_offline, topology/shutdown_push, licensing/apply, config/schema_update (dms-core), and security/sign, security/license_apply, security/ica_install (dms-ca).

Traces: dms-core, dms-ca and dms-opa-engine all export traces, always-sampled, to the endpoint configured via OTEL_EXPORTER_OTLP_ENDPOINT. There are no explicit spans around specific operations (licence issuance, node registration, config push); a span is instead created implicitly, once, per log call — the log message becomes the span name, and its purpose is purely to attach trace_id/span_id to that one log record, not to model an operation's duration. dms-opa-adapter and dms-crash-vault create no traces.

DMS-WebUX (browser-side): opt-in, two-layer gate — the deployment's own opt-in decision (DMS-Core's /v1/telemetry/optin, cached 15s) and a per-operator localStorage override. When on, @opentelemetry/sdk-trace-web instruments page loads and fetch() calls and posts spans to a same-origin relay (/api/telemetry/v1/traces, 1 MiB/batch cap, a scrubbing step strips user-agent/query-strings/headers first), which forwards them server-side to the deployment's own collector. The browser never talks to the collector or to FermiHDI directly, and the relay enforces the opt-in decision — a client cannot bypass it by editing browser storage.

Query Proxy (hd_proxy)

Optional: deploy it when you want a single mTLS/PBAC front door between clients and your Storage Clusters — see the front door page.

Metrics (/metrics, port 8080): a minimal set — fermihdi_queries_processed_total, fermihdi_bytes_received_total, fermihdi_bytes_sent_total (all counters). No records-scanned/selected, no PBAC-decision counter, no resource gauges.

Traces: real, operation-scoped spans, exported over OTLP to the DMS Collector — the inbound request handler (ServeHTTP), the OPA policy decision (PBAC_Evaluation), and the broadcast to Storage Cluster Controllers (SCC_Fanout) each get their own span. This is genuine per-operation instrumentation, unlike the incidental per-log-line span the DMS suite creates.

Logs: 2.0.0-shaped JSON, stdout only — like hd_sql/hd_graphql/ hd_mcp, these do not reach the DMS Collector. Unlike those three, hd_proxy now attaches a real tracing block on log lines emitted from inside its ServeHTTP/PBAC_Evaluation/SCC_Fanout spans — a genuine improvement: its own log lines can now be correlated to the traces those same spans send to the DMS Collector, even though the log lines themselves stay local.

Query services

These sit behind the Proxy on the client-facing side, each exposing a different wire protocol onto the same query engine. First, the naming: hd_sql, hd_graphql, hd_mcp and hd_data_explore are four distinct services, not overlapping ones — an empty hd_graphql/ subdirectory inside the hd_sql repo is a reserved/unused path and unrelated to the top-level hd_graphql repo. All four are optional: deploy the ones your clients need.

hd_sql, hd_graphql, and hd_mcp write their logs locally only — none of the three are wired into the DMS Collector (see Logging format above). None of them carry an OpenTelemetry SDK, so they never generate real trace spans; the tracing block in their logs only ever appears when a caller hands them a trace/span ID it already has.

If you're building a dashboard across "all query services," only hd_sql has real per-query metrics; treat the others' /metrics output as process-health only (or, for hd_data_explore, not present at all).

Data Explore (browser-side): opt-in, following the same shape as DMS-WebUX's telemetry relay but with a simpler gate — FERMIHDI_OPT_IN in this service's own server environment, not a DMS-Core API call (Data Explore has no mTLS client identity of its own to call DMS-Core with). When on, @opentelemetry/sdk-trace-web instruments page loads and fetch() calls and posts spans to a same-origin relay (/api/telemetry/v1/traces, 1 MiB/batch cap, the same scrubbing convention as DMS-WebUX's relay: no user-agent, query strings, or headers), which forwards them server-side to the deployment's own collector — the browser never talks to the collector directly. A structuredLog() helper emits the same 2.0.0 JSON envelope to the browser console.

Data Explore crash reporting: window.onerror/onunhandledrejection build a Sentry-format envelope (the crash vault's intake wire format, distinct from the logging_format_version JSON schema by design — see hd_dms/docs/crash_vault_openapi.yaml) and POST it to a second relay (/api/crash-report), which forwards it to the deployment's crash vault. Unlike the traces relay, this one is not gated by opt-in: the vault's intake always accepts and holds a report; opt-in only controls whether the vault later forwards a stack-only tier to FermiHDI on its own, a decision the vault makes, not this relay. This is genuinely new: no browser- originated crash-reporting path existed anywhere in this product before it.

Mandatory-metric coverage, by service

Before wiring up an alert against the "mandatory metrics" list in .agents/rules/observability-standard.md, check whether the service you're targeting actually publishes that metric — coverage varies by design (a control-plane service like DMS has no natural records_scanned, for example), not by omission everywhere:

• queries_processed: SN (fermihdi_query_sn_*), SCC (fanout snapshots), hd_sql, and hd_proxy publish this. hd_graphql, hd_mcp, DMS, BloomRF, and the Message Bus Ingester do not.
• records_stored/records_scanned/records_selected: SN publishes all three. hd_sql defines all three but only increments two (records_stored_total stays zero — the SQL server is read-only). No other service publishes these.
• bytes_received/bytes_sent: SCC, SN, hd_proxy, and hd_sql publish these. The Message Bus Ingester tracks them in-memory but does not expose them for scraping. BloomRF, DMS, hd_graphql, and hd_mcp do not publish these.
• cpu_utilization/memory_usage/disk_utilization: SN publishes memory_usage and disk_utilization as byte gauges (not percentages) — fermihdi_kv_memory_bytes/fermihdi_kv_disk_bytes. cpu_utilization is a dev-build-only PMU sample, compiled out of production builds. DMS defines a memory_usage_bytes series that is never set. No service publishes host disk or CPU utilization as a percentage.
• Traces: by design, the storage cluster (SCC/SN) only creates spans on the control-plane query path, never on ingest. Among the Go services, only hd_proxy creates real operation-scoped spans; DMS's spans are a side effect of its logger rather than independent instrumentation; hd_sql, hd_graphql, and hd_mcp create no spans.