HD SDK — Connecting, Wiring Up and Initialising

The HD SDK is a client library. You link it into your process; it does not run as a service, does not listen for HTTP, and exposes no REST API of its own. It gives you two classes:

Everything the SDK talks to is documented elsewhere, because it belongs to the service that serves it: registration in the DMS southbound API, client topology and dataset schema in the Proxy API, and the wire protocol in the FHDWP network guide.

Build and link

One public header, one shared library target:

#include <FermiHDISDK.hpp>

using namespace fermihdi::sdk;

target_link_libraries(your_app PRIVATE fermihdi_client)

fermihdi_client pulls in Apache Arrow, so anything appearing in the SDK's signatures as arrow::RecordBatch or arrow::Table is a normal Arrow object in your process — no copying or conversion at the boundary.

What construction actually does

Both clients boot the HD engine in their constructor. There is no separate init() or start() call; by the time the constructor returns, the engine thread is running and the client is ready to use.

Constructing a client will:

1. Probe for a GPU runtime and log which one it found, falling back to CPU routines when there is none.
2. Start the HD engine on its own thread, reserving one core and 1 GB of memory for it.
3. Bind the data plane to the interface you named.
4. Block until the engine reports ready. Expect the constructor to take a moment, and do not construct one per record — construct once and keep it.

Because a client owns a core and a gigabyte, treat it as a long-lived object. One per process is the normal shape.

Interfaces are yours to supply

FermiHDIIngestClient ingest("eth1", "eth0");
//                           ^data   ^control

FermiHDIQueryClient  query("https://proxy.internal:8443", "eth1", "eth0");
//                          ^proxy URL                     ^data   ^control

The SDK does not read environment variables to discover its interfaces. If you pass an empty string for the data plane interface it binds to all interfaces (0.0.0.0) rather than consulting the environment. On a host with separate data and control networks, always name them explicitly.

Two environment variables are read, and only to select the network stack:

Passing an empty proxy_url to FermiHDIQueryClient selects direct SCC mode: the client registers with the DMS as an HD_CLIENT node through register_with_dms(), which returns its certificate, the schema and the SCC targets, and it submits to those SCCs itself instead of to a Proxy. The SDK overview says when to use which; direct mode needs a registration token from an administrator and an HD_CLIENT licence element (or the HD_ANY pool).

What has to be reachable

That last row is the one that catches people. FermiHDIQueryClient opens a listener on port 6987 to receive result batches directly from Storage Nodes. If your firewall or security group blocks inbound 6987, queries will dispatch successfully and then hang until they time out, because the results have nowhere to land. Port 6987 is the FHDWP data plane port described in the FHDWP network guide.

Control-plane calls use mutual TLS with the certificate obtained during registration.

Wiring up the ingest path

The order matters. Each step depends on the one before it.

#include <FermiHDISDK.hpp>
using namespace fermihdi::sdk;

// 1. Construct. The engine boots here.
FermiHDIIngestClient client("eth1", "eth0");

// 2. Register with the DMS. Returns the client key in PEM form and stores
//    the signed certificate for later control-plane calls.
std::string key_pem = client.register_with_dms(
    "https://dms.internal:8443", "edge-collector-01");

// 3. Find the Storage Cluster Controllers, either by asking the proxy...
if (!client.discover_scc_targets("https://proxy.internal:8443")) {
    // ...or by naming them directly.
    client.set_scc_targets({{"10.0.4.11", 6987}, {"10.0.4.12", 6987}});
}

// 4. Load the dataset schema. Without it, record encoding produces nothing.
client.load_dataset_schema(schema_json);

// 5. Choose a batch ID. Everything in one batch shares it.
BatchID batch = client.generate_batch_id_time(start_epoch, end_epoch);

// 6. Encode and send.
for (const auto& row : rows) {
    const std::vector<uint8_t>& record = client.convert_record_to_binary(row);
    if (record.empty()) continue;          // no schema, or nothing matched
    client.stream_insert(batch, record);
}

// 7. Close the batch. Until the EOB lands, the storage layer holds it open.
client.send_eob(batch);

Steps 3 and 4 are both mandatory. With no SCC targets set, stream_insert(), send_eob() and send_bbn() all return false without sending anything. With no schema loaded, convert_record_to_binary() returns an empty buffer.

If you discover mid-batch that the data was bad, send a Bad Batch Notice instead of an EOB:

client.send_bbn(batch);   // tells the storage layer to discard the batch

Wiring up the query path

FermiHDIQueryClient client("https://proxy.internal:8443", "eth1", "eth0");

// Registration is only needed when querying SCCs directly (no proxy).
// It returns the full topology and the mTLS material in one object.
RegistrationResult reg = client.register_with_dms(
    "https://dms.internal:8443", "analytics-01");
if (reg.success) {
    client.set_scc_targets(reg.scc_targets);
    client.load_dataset_schema(reg.schema_json);
}

// Guard against a query that never completes.
client.set_query_timeout_ms(30'000);

std::shared_ptr<arrow::Table> table = client.execute_dataset(query_json);
// execute_stream() is available too, but note it completes the whole query
// before invoking your callback — it is not an incremental stream.

// A query can succeed and still be incomplete. Always check.
if (client.had_errors()) {
    for (const auto& err : client.get_batch_errors()) {
        // this batch was not delivered; the table is missing its records
    }
}

Two things worth getting right

Set a timeout. set_query_timeout_ms(0) is the default and means no timeout: if a Storage Node never returns its end-of-batch marker, the call blocks forever. When the timeout does fire, the SDK stops waiting and returns an empty table — not an error, and not a partial result.

It also sends POST /v1/query/stop to every known SCC target, but the SCC retired that route on 2026-08-09 and answers 410, so nothing downstream hears about it. The timeout and cancel_query() free your thread; the Storage Nodes carry on reading until the query finishes by itself. Size timeouts with that in mind — an aggressive one abandons work that still runs.

A successful query can be missing data. If a Storage Node sends a Bad Batch Notice instead of the records, the query completes normally and the Arrow Table silently omits those rows. had_errors() is the only thing that tells you, and get_batch_errors() says which batches went missing — one entry per failed batch, not one per notice received.

Removing the dispatch race

If you dispatch the query yourself and only then hand the SDK the expected batch list, end-of-batch markers can arrive before the SDK knows to expect them. Pre-arm the listener to avoid it:

client.prepare_query(query_id);          // BEFORE dispatching to the proxy
// ... dispatch, wait for the proxy to return found_batches ...
client.set_expected_batches(found_batches);   // AFTER
client.set_filter_verdict(would_exclude_batches, bloomrf_mode);  // from the same event
auto table = client.execute_dataset(query_json);

set_filter_verdict() matters on a cluster with a filter service: the SDK checks, when the query completes, that no batch the filter said could not match delivered a record. It only knows that list when it placed the query itself, so a self-dispatching caller passes it on. Skip it and the check silently never runs.

prepare_query() resets the query state and arms the listener, so markers arriving during the wait are kept instead of discarded. set_expected_batches() then completes the query immediately if everything already arrived. Skipping both still works — the SDK falls back to counting batches out of the query payload — but it reintroduces the race.

To stop an in-flight query from another thread:

client.cancel_query(query_id);   // safe to call from any thread

Health and metrics are strings, not endpoints

std::string health  = client.get_health_status();  // JSON
std::string metrics = client.get_metrics();        // Prometheus text format

The SDK does not serve these. It hands you the strings and your process decides what to do with them — typically publishing them on whatever health and metrics endpoints your application already exposes. If you want /healthz and /metrics for a process embedding the SDK, you serve them; the SDK has no listener to attach them to.

Shutting down

client.stop();     // stops the engine thread

The destructor also stops the client, so an object going out of scope is a clean shutdown. Call stop() explicitly when you need the core and the memory released at a point you control.

Send an EOB for any batch still open before stopping. A batch left open is a batch the storage layer is still waiting on.

When something returns false or null

Methods that are not implemented yet

Four public methods currently do nothing. They are declared in the header and they compile, but there is no behaviour behind them:

Do not build on these. To send Arrow data today, use the stream_insert(BatchID, std::shared_ptr<arrow::RecordBatch>, bool) overload, which is implemented, and construct the RecordBatch with the Arrow API directly.