Querying HD

Every query into an HD instance is the same shape underneath: a set of batches to read, a list of predicates the Storage Nodes evaluate before sending anything, and an address the records stream back to. That is the right interface for a store that moves records at line rate and the wrong one for the tools people already have, so HD offers the same query through several front doors. This page is the map: which door to use, what they have in common, and the three rules every one of them obeys.

The front doors

All of them run outside the storage cluster and reach it through the HD Proxy, which authenticates the caller, applies policy and scatters the query to every Storage Cluster Controller. The records do not come back through the Proxy: the Storage Nodes stream them straight to the front door, which evaluates the rest of the query locally with the analytical engine embedded in the SDK. The SQL, GraphQL and MCP services are therefore ordinary services to place — behind the same authenticating front door as the Proxy, on a host that the Storage Nodes can reach on the data plane — and the Administrator's Guide has their ports.

The SDK alone has a second way in. A client built on it can skip the Proxy and register with the DMS as an HD_CLIENT node: it presents a registration token an administrator issued, receives a certificate from the DMS-CA, the dataset schema and the list of Storage Cluster Controllers, and submits its queries to those controllers itself. Such a client is inside the trust boundary — the Proxy's policy does not apply to it, the DMS's admission does — and it draws on the HD_CLIENT element of the licence, or the HD_ANY pool. This is the mode for a service that lives inside the deployment, and for a custom ingester built on the SDK, which joins a cluster the same way. It is available from C++, Python and (for querying) JavaScript; the bindings built on the C ABI work in proxy mode only. The SDK overview puts the two modes side by side. What that front door is, what it authenticates and which identity each door carries to the Proxy is on the front doors page.

What every door has in common

The time window decides the cost. Records are stored in batches indexed by time, and a query names the batches it wants. A predicate on the dataset's batch-id field — usually the timestamp — becomes the window; a query that gives no window is either refused or answered over a default window that the service tells you about. Bound both ends.

Predicates the Storage Nodes can evaluate keep records off the network. Equality, not-equal, greater and less, on a declared field, against a literal. Everything else — >=, arithmetic, NOT, functions — is evaluated at the front door after the records arrive. The answer is the same either way; only the volume transferred differs, because the front door re-applies the whole condition to what it receives. A partial pushdown makes a query slower, never wrong.

Sixteen branches. A Storage Node evaluates at most sixteen alternative predicate sets per query. An OR is one branch per side and an IN list one per value; past sixteen the door pushes nothing down and reads the whole window, and says so. A range is two predicates where a list would be twenty.

A query lasts as long as its connection

Records come back out of band: the Storage Nodes stream them straight to the front door, and nothing on that path tells the cluster whether anyone is still listening. The caller's control connection is what carries that signal, so a query lives exactly as long as that connection stays open. Two consequences follow, and the bundled front doors and the SDK already honour both:

• Hold the connection open until every batch has arrived. Closing it as soon as the query is accepted tells the cluster the caller is done, and the Storage Nodes stop before they have sent anything. A query is not a fire-and-forget submission followed by a separate collection; the same connection that starts it must stay open to receive it.
• To stop early, cancel first. A caller that abandons a running query sends a cancel and then closes. A connection that simply drops is treated as the caller having gone, and the query is stopped on every Storage Node it reached. A clean close with no cancel means the caller received everything it wanted.

A query is never ended on elapsed time alone: a legitimate query may run for a long time, and the cluster distinguishes a large query from an abandoned one by the connection, not by a clock.

Addresses

An ipv4 field is a 32-bit unsigned integer and an ipv6 field a 128-bit one; that is what the ingester stores, what the Storage Nodes compare, and what the SDK's engine sees. Every front door converts at its own boundary and nowhere else: a literal written as an address — '10.0.0.1', '2001:db8::1' — becomes the integer before the query leaves the service, and a result column the schema calls an address is rendered as one on the way out. An IPv4 address against a sixteen-byte field is IPv4-mapped. The same conversion applies to a plain integer column that holds addresses, provided it is wide enough — four bytes for an IPv4, sixteen for an IPv6.

Ordering and comparison happen on the integer, so a range on an address column is numeric — 9.0.0.1 sorts before 10.0.0.1 — and only the presented value is text. The SDK translates nothing: a service or application built on it directly passes the integer and renders the result itself.

Empty windows and excluded batches

A window can hold nothing — no batch was stored for it, or, on a cluster with a filter node, every batch was proven unable to match. An aggregate over such a window still returns its row: a count is zero, a sum is null. A plain selection returns no rows.

The Proxy's answer to every query is a manifest rather than the records: which batches were found, which were missing, which a filter node excluded, and whether any Storage Cluster Controller timed out. The SQL, GraphQL and MCP services read it for you. A caller on the SDK reads it itself, and should hand the filter's verdict to the SDK so its false-negative check runs — a batch the filter excluded that delivers a record anyway is a defect the client is the only one positioned to see.

Choosing

Use the SQL server whenever the consumer already speaks SQL: it is the widest door and the one most tools reach without code. Use the IP-flow GraphQL service for the flow dataset specifically — its time series and top-N shapes are what a dashboard wants and would be several statements in SQL. Use the generic GraphQL service for an application that wants a typed schema over any dataset. Put an assistant on the MCP server rather than on raw SQL: its tools bound the time window and acknowledge the cost of a full scan on the caller's behalf. Reach for the SDK when you need the raw Arrow table, streaming record batches as they land, vector search or graph analysis over the result, or a language none of the services speak.