Querying HD¶
Every query into an HD instance is the same shape underneath: a set of batches to read, a list of predicates the Storage Nodes evaluate before sending anything, and an address the records stream back to. That is the right interface for a store that moves records at line rate and the wrong one for the tools people already have, so HD offers the same query through several front doors. This page is the map: which door to use, what they have in common, and the three rules every one of them obeys.
The front doors¶
| Door | Speaks | For | Page |
|---|---|---|---|
SQL server (hd_sql) |
PostgreSQL wire (5432), Arrow Flight SQL (31337), HTTP (8081) | BI tools, notebooks, JDBC and ODBC pipelines, anything with a database connector | SQL |
GraphQL (hd_graphql) |
GraphQL over HTTP | applications, with a schema generated from every dataset the Proxy serves | GraphQL |
IP-flow GraphQL (hd_ipflow_graphql) |
GraphQL over HTTP | dashboards and automation over the network-flow dataset: time series, top-N, aggregates | GraphQL |
MCP server (hd_mcp) |
the Model Context Protocol, over Server-Sent Events | AI assistants and agents that query on a person's behalf | MCP |
| Data Explore | a browser, on port 4321 | analysts, with a visual query builder and charts over the MCP server | — |
| The SDK | C++, Go, Python, JavaScript, Rust, Julia, R, Scala, Mojo | your own service, when none of the above fits | SDK overview |
All of them run outside the storage cluster and reach it through the HD Proxy, which authenticates the caller, applies policy and scatters the query to every Storage Cluster Controller. The records do not come back through the Proxy: the Storage Nodes stream them straight to the front door, which evaluates the rest of the query locally with the analytical engine embedded in the SDK. The SQL, GraphQL and MCP services are therefore ordinary services to place — behind the same authenticating front door as the Proxy, on a host that the Storage Nodes can reach on the data plane — and the Administrator's Guide has their ports.
The SDK alone has a second way in. A client built on it can skip the Proxy
and register with the DMS as an HD_CLIENT node: it presents a registration
token an administrator issued, receives a certificate from the DMS-CA, the
dataset schema and the list of Storage Cluster Controllers, and submits its
queries to those controllers itself. Such a client is inside the trust
boundary — the Proxy's policy does not apply to it, the DMS's admission does
— and it draws on the HD_CLIENT element of the licence, or the HD_ANY
pool. This is the mode for a service that lives inside the deployment, and
for a custom ingester built on the SDK, which joins a cluster the same way.
It is available from C++, Python and (for querying) JavaScript; the bindings
built on the C ABI work in proxy mode only. The
SDK overview
puts the two modes side by side. What that
front door is, what it authenticates and which identity each door carries to
the Proxy is on the front doors page.
What every door has in common¶
The time window decides the cost. Records are stored in batches indexed by time, and a query names the batches it wants. A predicate on the dataset's batch-id field — usually the timestamp — becomes the window; a query that gives no window is either refused or answered over a default window that the service tells you about. Bound both ends.
Predicates the Storage Nodes can evaluate keep records off the network.
Equality, not-equal, greater and less, on a declared field, against a literal.
Everything else — >=, arithmetic, NOT, functions — is evaluated at the
front door after the records arrive. The answer is the same either way; only
the volume transferred differs, because the front door re-applies the whole
condition to what it receives. A partial pushdown makes a query slower, never
wrong.
Sixteen branches. A Storage Node evaluates at most sixteen alternative
predicate sets per query. An OR is one branch per side and an IN list one
per value; past sixteen the door pushes nothing down and reads the whole
window, and says so. A range is two predicates where a list would be twenty.
A query lasts as long as its connection¶
Records come back out of band: the Storage Nodes stream them straight to the front door, and nothing on that path tells the cluster whether anyone is still listening. The caller's control connection is what carries that signal, so a query lives exactly as long as that connection stays open. Two consequences follow, and the bundled front doors and the SDK already honour both:
- Hold the connection open until every batch has arrived. Closing it as soon as the query is accepted tells the cluster the caller is done, and the Storage Nodes stop before they have sent anything. A query is not a fire-and-forget submission followed by a separate collection; the same connection that starts it must stay open to receive it.
- To stop early, cancel first. A caller that abandons a running query sends a cancel and then closes. A connection that simply drops is treated as the caller having gone, and the query is stopped on every Storage Node it reached. A clean close with no cancel means the caller received everything it wanted.
A query is never ended on elapsed time alone: a legitimate query may run for a long time, and the cluster distinguishes a large query from an abandoned one by the connection, not by a clock.
Addresses¶
An ipv4 field is a 32-bit unsigned integer and an ipv6 field a 128-bit one;
that is what the ingester stores, what the Storage Nodes compare, and what the
SDK's engine sees. Every front door converts at its own boundary and nowhere
else: a literal written as an address — '10.0.0.1', '2001:db8::1' — becomes
the integer before the query leaves the service, and a result column the
schema calls an address is rendered as one on the way out. An IPv4 address
against a sixteen-byte field is IPv4-mapped. The same conversion applies to a
plain integer column that holds addresses, provided it is wide enough — four
bytes for an IPv4, sixteen for an IPv6.
Ordering and comparison happen on the integer, so a range on an address column
is numeric — 9.0.0.1 sorts before 10.0.0.1 — and only the presented value
is text. The SDK translates nothing: a service or application built on it
directly passes the integer and renders the result itself.
Enumerations¶
A dataset may declare an enums table: named lists of labels, each assigned
an unsigned integer code, with one label as the default. A uint8, uint16,
uint32 or uint64 field that names one stores the code, and only the code;
the labels exist in the schema alone. Unlike an address, the code means
nothing without that table, so here the SDK does translate, on the strength
of what the schema declares: a result column comes back as its labels, text,
with a code the table does not define rendered as the default label, and a
filter on the field takes a label (or a defined code) and pushes the code to
the Storage Nodes, which see only the integer.
Two consequences follow. WHERE proto = 'tcp' and GROUP BY proto read
naturally in every front door, and ordering an enumerated column is by label,
not by code. And a label the table does not define is not pushed at all: the
default is for data, not predicates, so a mistyped label over-fetches and the
engine then matches nothing, rather than quietly selecting the "unknown"
rows. An application built on the SDK that wants the codes themselves turns
label decoding off on its client; the filter behaviour is unchanged by that.
Time fields and intra-record time compression¶
An integer field may declare time with a unit, nanoseconds through days:
its value is a count of that unit since the Unix epoch. A field carrying
time_reference instead stores a signed delta, in a unit of its own, from
another time field of the same record, so a handful of bytes hold what an
absolute time needs eight for. The schema declares the compression, so here
too the SDK translates: a result column comes back as the absolute time, a
64-bit integer in the field's own time unit or, when it declares none, the
source field's. On an array of integers every element expands against the
same source. An application built on the SDK that wants the stored deltas
turns expansion off on its client.
A filter on such a field is the one thing that is not translated: its value
is compared with the stored delta, because the absolute time it stands for
differs from record to record and there is no single byte pattern for the
Storage Nodes to match. Filter on the source time field for a time range,
and on the delta field for "within so many units of it". The SQL server is
the exception: a WHERE on such a column is evaluated there on the presented
time, never pushed down (see SQL server).
Chess notation¶
A chess_move field is stored as a 16-bit code that stands for one Standard
Algebraic Notation move, a chess_fen field as a Forsyth-Edwards position
packed into 32 bytes, and a chess_move_cc field as chess.com's own real
per-ply move code -- two raw bytes, exactly as chess.com's own Parquet
export stores them (see Ingestion for the layout). Unlike
chess_move, chess_move_cc has no formula: its code is a
reverse-engineered lookup table, not something computed from the move's
parts, so a raw value only means something through that table, and a code
not yet in it decodes to nothing -- the field still stores it losslessly.
chess_move and chess_fen render as their notation or as their raw stored
form according to the field's own render property in the schema ("raw"
or "decoded", default "decoded") -- not a setting an application turns
on or off on its client. Decoded, a result column of moves is the SAN text
(Nf3, exd5+, O-O), an array of moves a list of them, and a position
the full FEN string. Raw, a move column is the stored code (a plain integer)
and a position column the 32 packed bytes -- the form
fen_occupied/fen_turn filter pushdown already operates on directly, so
a dataset declaring chess_fen as raw is a real performance choice, not
just an alternate rendering.
chess_move_cc is different: every query service always hands it back raw
-- chess.com's own 2-character code, exactly as stored, never reinterpreted
as a number -- regardless of render (a field may still declare it, for
forward compatibility, but it has no effect). Decoding it to long
algebraic notation (e2e4, g1f3, e7e8q -- not SAN; check and mate are
not part of chess.com's encoding) is an explicit capability a consumer
calls itself against the codebook: hd_sdk's chess_cc::decode_chess_cc()
in C++, or the equivalent client-side decode Data Explore's chess board
graph does against /api/chess-cc-codebook. No query service does this
decode on a caller's behalf.
A filter takes the notation too: an EQUALITY on a chess_move field with
Nf3 matches the records whose stored code is that move's, one on a
chess_fen field with a FEN string matches the records holding exactly
that position, and one on a chess_move_cc field matches its own raw
2-character code -- there is no way to filter it by long algebraic notation,
since encoding LAN back into chess.com's code would need the reverse of a
lookup table that only decodes today.
Beyond a whole position, the packed layout lets a few questions about a
position be asked of the Storage Nodes directly, as a masked comparison on
the bytes: whether a square is occupied (one bit of the occupancy mask) and
which side is to move (one bit of the game state). The SQL server pushes
fen_occupied(position, 'e4') and fen_turn(position) = 'b' that way, and
evaluates the rest of the fen_* functions itself: fen_piece(position,
'e4') (the piece letter, empty for an empty square), fen_castling,
fen_en_passant, fen_halfmove, fen_fullmove and fen_piece_count, plus
san_code and san_text to move between a SAN string and its code. The
GraphQL and MCP servers reach the same functions through their SQL paths.
chess_move_cc has no equivalent function set today -- an equality filter
on the raw code (above) is the whole of what the query path itself offers;
decoding is a client-side capability, not a query-side one.
chess_move_cc's decode table is reverse-engineered from chess.com's own
real data, not a published chess.com specification, and covers standard
chess only (no Chess960, bughouse, crazyhouse, three-check,
king-of-the-hill or odds chess). It has no guarantee of staying unchanged
in a future chess.com export.
Data Explore draws a game. Its Chess Board graph takes a position column as the starting board and a moves column as the game to replay from it, either one alone working too: a position on its own is shown as stored, moves on their own replay from the standard start. Pick a row, step through the plies or let them play, and the board follows, with the last move highlighted and the position's FEN beneath it. A move that is not legal where it stands ends the replay and is named.
Arrays¶
A bytes field may declare an array: an element type, how many elements the
field holds at most, and for string or numeric elements the width of each.
The field then holds a count followed by the elements, each stored exactly as
a field of that type would be, with the room past the count left zero, so a
record's bytes are still a function of its value alone. The field's type
stays bytes, so a consumer that never reads the declaration still gets the
bytes it was promised.
On the query path the SDK hands the field back as a list of the element type,
and every front door follows: a DuckDB LIST in the SQL server (integer[]
in information_schema, a PostgreSQL array on the wire), a list in GraphQL,
a JSON array from the MCP server and in Data Explore. An enum on the field
labels each element.
A predicate on an array field asks whether any element satisfies it, and it
is pushed to the Storage Nodes as such. The SDK turns tags = 8 into an
element group the storage node walks over the stored elements, matching a
record when some element equals 8; != matches when no element equals the
value; > and < when some element does. In SQL that is list_contains(tags,
8) and NOT list_contains(tags, 8), in GraphQL and the MCP server the same
EQUALITY, NOT_EQUALITY, GREATER and LESS operators as on a scalar
field, with a label where the elements carry an enum. A filter node indexes
an array by its elements, so a contains predicate prunes batches the way a
scalar equality does. An element wider than eight bytes (an ipv6 or a wide
string) is the one case the storage node cannot walk, and a predicate on such
a field is evaluated after the fetch, with a notice where the front door has
one.
Empty windows and excluded batches¶
A window can hold nothing — no batch was stored for it, or, on a cluster with a filter node, every batch was proven unable to match. An aggregate over such a window still returns its row: a count is zero, a sum is null. A plain selection returns no rows.
The Proxy's answer to every query is a manifest rather than the records: which batches were found, which were missing, which a filter node excluded, and whether any Storage Cluster Controller timed out. The SQL, GraphQL and MCP services read it for you. A caller on the SDK reads it itself, and should hand the filter's verdict to the SDK so its false-negative check runs — a batch the filter excluded that delivers a record anyway is a defect the client is the only one positioned to see.
Choosing¶
Use the SQL server whenever the consumer already speaks SQL: it is the widest door and the one most tools reach without code. Use the IP-flow GraphQL service for the flow dataset specifically — its time series and top-N shapes are what a dashboard wants and would be several statements in SQL. Use the generic GraphQL service for an application that wants a typed schema over any dataset. Put an assistant on the MCP server rather than on raw SQL: its tools bound the time window and acknowledge the cost of a full scan on the caller's behalf. Reach for the SDK when you need the raw Arrow table, streaming record batches as they land, vector search or graph analysis over the result, or a language none of the services speak.