Querying HD

Every query into an HD instance is the same shape underneath: a set of batches to read, a list of predicates the Storage Nodes evaluate before sending anything, and an address the records stream back to. That is the right interface for a store that moves records at line rate and the wrong one for the tools people already have, so HD offers the same query through several front doors. This page is the map: which door to use, what they have in common, and the three rules every one of them obeys.

The front doors

All of them run outside the storage cluster and reach it through the HD Proxy, which authenticates the caller, applies policy and scatters the query to every Storage Cluster Controller. The records do not come back through the Proxy: the Storage Nodes stream them straight to the front door, which evaluates the rest of the query locally with the analytical engine embedded in the SDK. The SQL, GraphQL and MCP services are therefore ordinary services to place — behind the same authenticating front door as the Proxy, on a host that the Storage Nodes can reach on the data plane — and the Administrator's Guide has their ports.

The SDK alone has a second way in. A client built on it can skip the Proxy and register with the DMS as an HD_CLIENT node: it presents a registration token an administrator issued, receives a certificate from the DMS-CA, the dataset schema and the list of Storage Cluster Controllers, and submits its queries to those controllers itself. Such a client is inside the trust boundary — the Proxy's policy does not apply to it, the DMS's admission does — and it draws on the HD_CLIENT element of the licence, or the HD_ANY pool. This is the mode for a service that lives inside the deployment, and for a custom ingester built on the SDK, which joins a cluster the same way. It is available from C++, Python and (for querying) JavaScript; the bindings built on the C ABI work in proxy mode only. The SDK overview puts the two modes side by side. What that front door is, what it authenticates and which identity each door carries to the Proxy is on the front doors page.

What every door has in common

The time window decides the cost. Records are stored in batches indexed by time, and a query names the batches it wants. A predicate on the dataset's batch-id field — usually the timestamp — becomes the window; a query that gives no window is either refused or answered over a default window that the service tells you about. Bound both ends.

Predicates the Storage Nodes can evaluate keep records off the network. Equality, not-equal, greater and less, on a declared field, against a literal. Everything else — >=, arithmetic, NOT, functions — is evaluated at the front door after the records arrive. The answer is the same either way; only the volume transferred differs, because the front door re-applies the whole condition to what it receives. A partial pushdown makes a query slower, never wrong.

Sixteen branches. A Storage Node evaluates at most sixteen alternative predicate sets per query. An OR is one branch per side and an IN list one per value; past sixteen the door pushes nothing down and reads the whole window, and says so. A range is two predicates where a list would be twenty.

A query lasts as long as its connection

Records come back out of band: the Storage Nodes stream them straight to the front door, and nothing on that path tells the cluster whether anyone is still listening. The caller's control connection is what carries that signal, so a query lives exactly as long as that connection stays open. Two consequences follow, and the bundled front doors and the SDK already honour both:

• Hold the connection open until every batch has arrived. Closing it as soon as the query is accepted tells the cluster the caller is done, and the Storage Nodes stop before they have sent anything. A query is not a fire-and-forget submission followed by a separate collection; the same connection that starts it must stay open to receive it.
• To stop early, cancel first. A caller that abandons a running query sends a cancel and then closes. A connection that simply drops is treated as the caller having gone, and the query is stopped on every Storage Node it reached. A clean close with no cancel means the caller received everything it wanted.

A query is never ended on elapsed time alone: a legitimate query may run for a long time, and the cluster distinguishes a large query from an abandoned one by the connection, not by a clock.

Addresses

An ipv4 field is a 32-bit unsigned integer and an ipv6 field a 128-bit one; that is what the ingester stores, what the Storage Nodes compare, and what the SDK's engine sees. Every front door converts at its own boundary and nowhere else: a literal written as an address — '10.0.0.1', '2001:db8::1' — becomes the integer before the query leaves the service, and a result column the schema calls an address is rendered as one on the way out. An IPv4 address against a sixteen-byte field is IPv4-mapped. The same conversion applies to a plain integer column that holds addresses, provided it is wide enough — four bytes for an IPv4, sixteen for an IPv6.

Ordering and comparison happen on the integer, so a range on an address column is numeric — 9.0.0.1 sorts before 10.0.0.1 — and only the presented value is text. The SDK translates nothing: a service or application built on it directly passes the integer and renders the result itself.

Enumerations

A dataset may declare an enums table: named lists of labels, each assigned an unsigned integer code, with one label as the default. A uint8, uint16, uint32 or uint64 field that names one stores the code, and only the code; the labels exist in the schema alone. Unlike an address, the code means nothing without that table, so here the SDK does translate, on the strength of what the schema declares: a result column comes back as its labels, text, with a code the table does not define rendered as the default label, and a filter on the field takes a label (or a defined code) and pushes the code to the Storage Nodes, which see only the integer.

Two consequences follow. WHERE proto = 'tcp' and GROUP BY proto read naturally in every front door, and ordering an enumerated column is by label, not by code. And a label the table does not define is not pushed at all: the default is for data, not predicates, so a mistyped label over-fetches and the engine then matches nothing, rather than quietly selecting the "unknown" rows. An application built on the SDK that wants the codes themselves turns label decoding off on its client; the filter behaviour is unchanged by that.

Time fields and intra-record time compression

An integer field may declare time with a unit, nanoseconds through days: its value is a count of that unit since the Unix epoch. A field carrying time_reference instead stores a signed delta, in a unit of its own, from another time field of the same record, so a handful of bytes hold what an absolute time needs eight for. The schema declares the compression, so here too the SDK translates: a result column comes back as the absolute time, a 64-bit integer in the field's own time unit or, when it declares none, the source field's. On an array of integers every element expands against the same source. An application built on the SDK that wants the stored deltas turns expansion off on its client.

A filter on such a field is the one thing that is not translated: its value is compared with the stored delta, because the absolute time it stands for differs from record to record and there is no single byte pattern for the Storage Nodes to match. Filter on the source time field for a time range, and on the delta field for "within so many units of it". The SQL server is the exception: a WHERE on such a column is evaluated there on the presented time, never pushed down (see SQL server).

Chess notation

A chess_move field is stored as a 16-bit code that stands for one Standard Algebraic Notation move, a chess_fen field as a Forsyth-Edwards position packed into 32 bytes, and a chess_move_cc field as chess.com's own real per-ply move code -- two raw bytes, exactly as chess.com's own Parquet export stores them (see Ingestion for the layout). Unlike chess_move, chess_move_cc has no formula: its code is a reverse-engineered lookup table, not something computed from the move's parts, so a raw value only means something through that table, and a code not yet in it decodes to nothing -- the field still stores it losslessly.

chess_move and chess_fen render as their notation or as their raw stored form according to the field's own render property in the schema ("raw" or "decoded", default "decoded") -- not a setting an application turns on or off on its client. Decoded, a result column of moves is the SAN text (Nf3, exd5+, O-O), an array of moves a list of them, and a position the full FEN string. Raw, a move column is the stored code (a plain integer) and a position column the 32 packed bytes -- the form fen_occupied/fen_turn filter pushdown already operates on directly, so a dataset declaring chess_fen as raw is a real performance choice, not just an alternate rendering.

chess_move_cc is different: every query service always hands it back raw -- chess.com's own 2-character code, exactly as stored, never reinterpreted as a number -- regardless of render (a field may still declare it, for forward compatibility, but it has no effect). Decoding it to long algebraic notation (e2e4, g1f3, e7e8q -- not SAN; check and mate are not part of chess.com's encoding) is an explicit capability a consumer calls itself against the codebook: hd_sdk's chess_cc::decode_chess_cc() in C++, or the equivalent client-side decode Data Explore's chess board graph does against /api/chess-cc-codebook. No query service does this decode on a caller's behalf.

A filter takes the notation too: an EQUALITY on a chess_move field with Nf3 matches the records whose stored code is that move's, one on a chess_fen field with a FEN string matches the records holding exactly that position, and one on a chess_move_cc field matches its own raw 2-character code -- there is no way to filter it by long algebraic notation, since encoding LAN back into chess.com's code would need the reverse of a lookup table that only decodes today.

Beyond a whole position, the packed layout lets a few questions about a position be asked of the Storage Nodes directly, as a masked comparison on the bytes: whether a square is occupied (one bit of the occupancy mask) and which side is to move (one bit of the game state). The SQL server pushes fen_occupied(position, 'e4') and fen_turn(position) = 'b' that way, and evaluates the rest of the fen_* functions itself: fen_piece(position, 'e4') (the piece letter, empty for an empty square), fen_castling, fen_en_passant, fen_halfmove, fen_fullmove and fen_piece_count, plus san_code and san_text to move between a SAN string and its code. The GraphQL and MCP servers reach the same functions through their SQL paths. chess_move_cc has no equivalent function set today -- an equality filter on the raw code (above) is the whole of what the query path itself offers; decoding is a client-side capability, not a query-side one.

chess_move_cc's decode table is reverse-engineered from chess.com's own real data, not a published chess.com specification, and covers standard chess only (no Chess960, bughouse, crazyhouse, three-check, king-of-the-hill or odds chess). It has no guarantee of staying unchanged in a future chess.com export.

Data Explore draws a game. Its Chess Board graph takes a position column as the starting board and a moves column as the game to replay from it, either one alone working too: a position on its own is shown as stored, moves on their own replay from the standard start. Pick a row, step through the plies or let them play, and the board follows, with the last move highlighted and the position's FEN beneath it. A move that is not legal where it stands ends the replay and is named.

Arrays

A bytes field may declare an array: an element type, how many elements the field holds at most, and for string or numeric elements the width of each. The field then holds a count followed by the elements, each stored exactly as a field of that type would be, with the room past the count left zero, so a record's bytes are still a function of its value alone. The field's type stays bytes, so a consumer that never reads the declaration still gets the bytes it was promised.

On the query path the SDK hands the field back as a list of the element type, and every front door follows: a DuckDB LIST in the SQL server (integer[] in information_schema, a PostgreSQL array on the wire), a list in GraphQL, a JSON array from the MCP server and in Data Explore. An enum on the field labels each element.

A predicate on an array field asks whether any element satisfies it, and it is pushed to the Storage Nodes as such. The SDK turns tags = 8 into an element group the storage node walks over the stored elements, matching a record when some element equals 8; != matches when no element equals the value; > and < when some element does. In SQL that is list_contains(tags, 8) and NOT list_contains(tags, 8), in GraphQL and the MCP server the same EQUALITY, NOT_EQUALITY, GREATER and LESS operators as on a scalar field, with a label where the elements carry an enum. A filter node indexes an array by its elements, so a contains predicate prunes batches the way a scalar equality does. An element wider than eight bytes (an ipv6 or a wide string) is the one case the storage node cannot walk, and a predicate on such a field is evaluated after the fetch, with a notice where the front door has one.

Empty windows and excluded batches

A window can hold nothing — no batch was stored for it, or, on a cluster with a filter node, every batch was proven unable to match. An aggregate over such a window still returns its row: a count is zero, a sum is null. A plain selection returns no rows.

The Proxy's answer to every query is a manifest rather than the records: which batches were found, which were missing, which a filter node excluded, and whether any Storage Cluster Controller timed out. The SQL, GraphQL and MCP services read it for you. A caller on the SDK reads it itself, and should hand the filter's verdict to the SDK so its false-negative check runs — a batch the filter excluded that delivers a record anyway is a defect the client is the only one positioned to see.

Choosing

Use the SQL server whenever the consumer already speaks SQL: it is the widest door and the one most tools reach without code. Use the IP-flow GraphQL service for the flow dataset specifically — its time series and top-N shapes are what a dashboard wants and would be several statements in SQL. Use the generic GraphQL service for an application that wants a typed schema over any dataset. Put an assistant on the MCP server rather than on raw SQL: its tools bound the time window and acknowledge the cost of a full scan on the caller's behalf. Reach for the SDK when you need the raw Arrow table, streaming record batches as they land, vector search or graph analysis over the result, or a language none of the services speak.