HD SDK

The client library for querying and ingesting against an HD storage cluster. C++ at its core, with bindings for eight languages.

All language wrappers are BETA. They build and link against a real SDK — that is verified on every change by ./build_wrappers.sh — but they have not been exercised against a live cluster in the way the C++ core and the Go binding have. Expect rough edges in the less-travelled ones, and read Wrapper maturity before choosing one for production.

The one thing to understand first

The SDK owns the data plane. It does not own the control plane.

A query is two separate conversations with two different pieces of the system, and almost every mistake people make with this SDK comes from thinking it is one.

  your process                     the cluster
  ┌────────────────────┐
  │  your code         │──── 1. control plane ───▶  Proxy ──▶ Storage Cluster
  │                    │      "run this query"              Controllers (SCC)
  │  ┌──────────────┐  │                                          │
  │  │   the SDK    │  │                                          │ fan-out
  │  │ FHDWP        │  │                                          ▼
  │  │ listener     │◀─┼──── 2. data plane ──────────────  Storage Nodes (SN)
  │  └──────────────┘  │      records, out of band
  └────────────────────┘

1. Control plane. A query is submitted over HTTPS to a Query Proxy, or over gRPC directly to storage cluster controllers. This is a small request that answers one question: which batches actually exist for what you asked?
2. Data plane. The Storage Nodes then stream the records straight to your process, to an address and port you advertised in the query. They do not travel back through the proxy. The SDK's job is to be listening.

Consequences that surprise people:

• Nothing arrives on the connection you submitted the query on. If you are waiting for records to come back from the HTTP response, you will wait forever.
• The address you advertise must be reachable from the storage nodes. Get it wrong and the query completes with zero records and no error — the nodes streamed perfectly, to nowhere. See data_plane_advertised_endpoint.
• The listener must be armed before you submit. Nodes can begin streaming the moment the controllers fan out, which is before your submission call returns. Records that arrive at an unarmed listener are discarded. This is what prepare_query() exists for, and why submit_query_to_proxy() calls it for you.

Two ways in: through the Proxy, or direct

The control-plane conversation has two forms, and which one you are in decides what the SDK needs before the first query.

Proxy mode is the default. The client submits to the Query Proxy through the customer's query front door; the front door authenticates the caller and the Proxy's policy applies to the query. The SDK discovers the SCC targets and the dataset schema from the Proxy. Nothing registers anywhere.

Direct mode is selected by constructing the query client with an empty proxy_url. The client then registers itself with the DMS as an HD_CLIENT node — register_with_dms() sends a certificate signing request and a registration token, and returns the certificate the DMS-CA signed, the private key, the dataset schema and the SCC targets — and submits queries over gRPC to those SCCs itself. An HD_CLIENT registration draws on the HD_CLIENT element of the licence, or on the HD_ANY pool when the licence has none, exactly as a deployed node does, and the DMS admits it under the same policy. The Proxy's policy does not apply: a direct client is inside the trust boundary, which is why it needs a registration token an administrator issued. A custom ingester built on FermiHDIIngestClient joins a cluster the same way: it registers as HD_CLIENT, receives the SCC targets, and sends its batches straight to them.

Both modes receive records the same way, over the FHDWP data plane on the interface the client names. Direct mode is for services that live inside the deployment, custom ingesters, and the cases where the Proxy's hop is not wanted; proxy mode is for everything a person or an external application does.

The C ABI carries fhd_query_client_set_scc_target but no registration call and no ingest client, so the six bindings built on it can be handed SCC targets but cannot register as HD_CLIENT or ingest. Until the C ABI gains both, direct mode and custom ingesters are C++, Python and, for querying, JavaScript.

The lifecycle of a query

   prepare_query(id)          arm the listener FIRST
          │
          ▼
   submit to the proxy        POST /v1/hd/query  ── control plane
          │                   ← found_batches / missing_batches
          ▼
   set_expected_batches()     wait for exactly what exists, not what you asked
   set_filter_verdict()       and what the filter service would have skipped
          │
          ▼
   execute_dataset()          blocks while records arrive on the DATA plane
   execute_to_sql()           returns an Arrow Table
          │
          ▼
   report the ending          complete, or cancel

The table is yours

Every execute call ends in an Arrow table, and nothing here hides it. In C++ execute_dataset() returns the raw record table (one FixedSizeBinary column of the dataset's record width, untouched), execute_to_sql() the projected one, execute_stream() the record batches as they land, and apply_sql() runs SQL over a table you already hold. Across the C ABI, Go and JS the same table travels as an Arrow IPC stream: execute_dataset_to_ipc (raw), execute_to_ipc (projected), apply_sql_to_ipc (yours in, result out). In Python the table is an ArrowTable with to_ipc(), which pyarrow.ipc.open_stream() reads, and table_from_ipc() brings one back for apply_sql(). Decode it, filter it, or hand it to your own engine: the SDK does not translate anything in it.

Why set_expected_batches matters

Your query names a window. The cluster holds whatever it holds. Ask for a day and it may have eleven hours of it.

The control plane answers with found_batches — the batches that exist — and missing_batches. If you wait for the batches you asked for, you wait for ones that were never stored, and the query hangs until its timeout with a partial result. set_expected_batches(found) is what makes a query finish promptly.

Endings, and why they are not optional

A query costs storage-node work for as long as it runs. Because the data plane is one-way, no storage node can tell whether anyone is still listening. A client that vanishes leaves nodes reading NVMe and streaming into a void.

So a query ends in exactly one of two ways:

The rule: a client that closes early MUST send a cancel first.

That rule is what lets a clean disconnect be read as "done" rather than as abandonment. It is written up in full in .agent/rules/query-lifecycle-standard.md.

The held stream

When you submit through a proxy, the SDK keeps that connection open for the entire life of the query and sends the ending on it. That connection is the only thing telling the cluster you are still there.

submit_query_to_proxy() does all of this — arms the listener, submits, feeds back the batch list, holds the connection, and reports the ending from the same place the completion and cancellation handlers fire. If you use it, endings are handled and you do not need to register a handler at all.

If you submit the query yourself (direct SCC mode, or your own HTTP client), you own the ending. Register set_cancel_handler() and set_complete_handler() and relay them.

Queries

Batch IDs

A batch is the unit of storage and of fan-out. A batch ID is 64 raw bytes, usually written as 128 hex characters. For time-partitioned datasets the ID encodes the bucket timestamp in its last 8 bytes, which is why the SDK can generate the IDs for a time range without asking anyone:

generate_batch_ids_time(from_epoch_sec, to_epoch_sec)

The full-dataset scan guard

An empty batch_ids list is an open query: every batch on every storage node. It is legitimate, and it is the most expensive thing the platform can do. The cost is invisible in the statement that produced it — a SELECT with no WHERE looks exactly like one with a narrow WHERE.

So it cannot be reached by omission:

generate_hd_query() refuses a query with no batch_ids
unless acknowledge_full_dataset_scan is set, and returns an empty string.

• A caller with a user interface should set it from an explicit acknowledgement — a checkbox, a confirmation — and never by default.
• A caller without one (a service, a batch job) sets it in code, having decided that a full scan is what it means.

If generate_hd_query() hands you an empty string, this is why.

Sub-queries and filters

Predicates are pushed down to the storage nodes. You express them as filters on named schema fields; the SDK compiles them into the mask/match byte strings the nodes evaluate.

Filter { field_name, value, op }        op: EQUALITY | NOT_EQUAL | GREATER | LESS
SubQuery { filters[] }                   filters within a sub-query are ANDed
HDQuery { sub_queries[] }                sub-queries are ORed

Limit: 16 sub-queries. Beyond that the storage node silently drops the excess. Restructure rather than exceed it.

Addressing

Two different addresses, and confusing them is the single most common cause of "the query returned nothing and said nothing":

They are the same on a flat network and different behind any NAT — a client in a container binds 6987 inside it while the nodes must be told the host's address and published port. Whatever you advertise must appear in the query's targets.

Two clients on one host cannot both bind the default 6987, so the port is a setting, not a constant. It takes effect only before the listener starts.

Health

get_health_status() answers two different questions, and they are kept apart deliberately:

{"status": "ok", "ready": false, "timestamp": 1788589254.13}

A client that has been constructed but whose engine has not booted — no schema yet, because the proxy was unreachable at init — is "ok" and not ready. It will boot on the first query. Restarting it would not have made the proxy reachable, so reporting it as unhealthy would be the wrong answer to the question a liveness probe is asking.

Building against the SDK

Getting it

The package image carries everything at /opt/fermihdi-sdk:

/opt/fermihdi-sdk/
├── include/          headers, including fermihdi_c_api.h
├── lib/              libfermihdi_client.so, libfermihdi_client_static.a
├── cgo-env.sh        the link line for Go, ready to source
├── go/ python/ rust/ js/ julia/ r/ scala/ mojo/
└── PACKAGE.txt       version, variant, provenance

COPY --from=cr.fermihdi.io/hd/sdk:<version> /opt/fermihdi-sdk /opt/fermihdi-sdk

The link line — read this before debugging a link error

libfermihdi_client.so embeds the HD engine's runtime through a static archive, and a shared object linked against a static archive does not re-export that archive's transitive dependencies. Linking -lfermihdi_client alone fails with undefined references to fmt, hwloc, NORM, boost, c-ares and yaml-cpp.

Every consumer must name them:

-lfermihdi_client -larrow -lnorm -lfmt -lhwloc -lyaml-cpp \
-lboost_program_options -lboost_thread -lcares -lcurl $(pkg-config --libs libdpdk)

For Go this is written for you: . /opt/fermihdi-sdk/cgo-env.sh.

Use -Wl,--no-as-needed ahead of that list if your own code does not reference those symbols directly — with the default --as-needed the linker records no DT_NEEDED entry, the link succeeds, and the failure moves to runtime as an undefined symbol.

Dynamically loaded consumers need static TLS surplus

This affects Python, R and Node — anything the host process dlopens.

The HD engine's runtime uses initial-exec thread-local storage. A library with initial-exec TLS cannot be dlopened into an already-running process unless the loader reserved surplus static TLS, and you get:

cannot allocate memory in static TLS block

Set the tunable before starting the host process:

GLIBC_TUNABLES=glibc.rtld.optional_static_tls=1048576 R -f script.R
GLIBC_TUNABLES=glibc.rtld.optional_static_tls=1048576 python3 app.py

Compiled consumers that link at build time (Go, Rust, Mojo, C++) are unaffected.

Static linking

libfermihdi_client_static.a is built beside the shared library (-DFERMIHDI_BUILD_STATIC=ON, the default). Link it instead of the .so when you want a binary you can copy to a host on its own, with no rpath to get right.

The C ABI

include/fermihdi_c_api.h exports a flat C surface from the shared library, so any language with FFI can use the SDK without a C++ toolchain. It is what the Mojo binding is built on, and it is the recommended base for a new binding.

FhdQueryClient* fhd_query_client_new(proxy_url, dp_iface, cp_iface, return_port);
void            fhd_query_client_free(client);
bool            fhd_query_client_load_schema(client, schema_json);
char*           fhd_query_client_generate_query(client, hdquery_json);
char*           fhd_query_client_submit_to_proxy(client, query_json, query_id);
const uint8_t*  fhd_query_client_execute_to_ipc(client, query_json, sql, &size);
void            fhd_query_client_cancel_query(client, query_id);

Conventions:

• Nothrow. A C++ exception crossing an FFI boundary is undefined behaviour, so every entry point catches and reports through its return value.
• char* returns are caller-owned — release with fhd_string_free; byte buffers with fhd_bytes_free.
• *_into variants fill a buffer you own and return the length needed, which avoids ownership transfer entirely. Prefer these in languages where a nullable pointer is awkward to model.
• A NULL client is tolerated everywhere, so code that failed to construct one cannot crash by carrying on.

Wrapper maturity

All wrappers are beta. Every one is built and linked against a real SDK by ./build_wrappers.sh, which runs in CI, so none of them is broken. What separates them is how much real use they have had.

Beta means: it builds, it links, its API is documented and exercised by a smoke build. It does not mean every call has been run against a live cluster. If you are choosing for production today, Go has by far the most mileage.

Verifying a change

Building the core and every wrapper, and the tests that guard them, are the developer guide's subject. A consumer of the published package never builds any of it.

Where to go next

• Language guides — the per-language READMEs linked in §7, each with a full API reference, a worked example and use cases.
• Query lifecycle rules — .agent/rules/query-lifecycle-standard.md.
• The C ABI — include/fermihdi_c_api.h.
• The C++ API — include/FermiHDISDK.hpp, documented inline.