HD SDK — API Reference

Every public type and method in <FermiHDISDK.hpp>, in namespace fermihdi::sdk. For how to put them together, see the getting started guide.

Both clients use the pimpl idiom, are non-copyable, and boot the HD engine in their constructor. See Not implemented for the four methods that currently do nothing.

Types

BatchID

using BatchID = std::array<uint8_t, 64>;

A fixed 64-byte batch identifier. Every record, end-of-batch marker and bad batch notice carries one. Build them with the generate_batch_id_* methods rather than by hand — the byte layout is part of the wire contract.

FilterOp

enum class FilterOp { EQUALITY, NOT_EQUALITY, GREATER, LESS };

Filter

struct Filter {
    std::string field_name;
    std::string value;
    FilterOp    op;
};

One predicate against one field. value is a string regardless of the field's declared type; it is interpreted according to the schema.

SubQuery

struct SubQuery { std::vector<Filter> filters; };

A group of filters evaluated together.

SccAddress

struct SccAddress { std::string ip; uint16_t port; };

One Storage Cluster Controller on the data plane.

HDQuery

struct HDQuery {
    uint32_t                query_id;
    std::vector<BatchID>    batch_ids;
    std::vector<SubQuery>   sub_queries;
    std::vector<SccAddress> targets;
    bool                    verify_hash = false;
    unsigned int            query_qos   = 1;
};

BatchError

struct BatchError { BatchID batch_id; std::string error; };

One batch that failed delivery during a query — typically because the Storage Node sent a bad batch notice instead of the data. See had_errors().

RegistrationResult

struct RegistrationResult {
    bool                    success;
    std::string             client_key_pem;
    std::string             signed_cert_pem;
    std::string             schema_json;
    std::vector<SccAddress> scc_targets;
    std::string             error_message;
};

Returned by FermiHDIQueryClient::register_with_dms(). One round trip yields the mTLS material, the dataset schema and the cluster topology, so a registered query client can be fully configured without further discovery calls.

FermiHDIIngestClient

The write path: encode records, stream them to SCCs, close the batch.

Construction and teardown

explicit FermiHDIIngestClient(const std::string& data_plane_interface = "",
                              const std::string& control_plane_interface = "");
~FermiHDIIngestClient();
void stop();

Boots the HD engine on its own thread, reserving one core and 1 GB, and blocks until it reports ready. Name your interfaces — an empty data plane interface binds all interfaces rather than reading the environment. Construct once and keep it; this is not a per-record object.

stop() stops the engine thread. The destructor does the same, so scope exit is a clean shutdown.

register_with_dms

std::string register_with_dms(const std::string& dms_url,
                              const std::string& node_name,
                              const std::string& override_key_pem = "");

Registers against the DMS, storing the signed certificate and client key for subsequent control-plane calls. Returns the client key in PEM form; an empty string means registration failed. Pass override_key_pem to register with a key you already hold instead of a freshly generated one.

discover_scc_targets

bool discover_scc_targets(const std::string& proxy_url);

Asks the proxy for the cluster topology and populates the SCC target list from the primary cluster it reports. Returns false if the proxy is unreachable, the response cannot be parsed, or it names no targets — check the return value, as a failure leaves you with no targets and every send will then fail.

add_control_header

void add_control_header(const std::string& key, const std::string& value);

Adds an HTTP header to every subsequent control-plane request, for proxy routing or tenancy. Headers accumulate; there is no method to clear them.

set_scc_targets

void set_scc_targets(const std::vector<SccAddress>& targets);

Sets the targets directly, replacing any discovered set. The first target is the primary and receives everything sent without broadcast, so ordering is significant.

load_dataset_schema

void load_dataset_schema(const std::string& schema_json);

Loads the schema describing the fixed-width record layout — field names, byte ranges, types, and the time index interval used by generate_batch_id_time(). Required before any record encoding.

convert_record_to_binary

const std::vector<uint8_t>& convert_record_to_binary(
    const std::map<std::string, std::string>& kv_pairs);

Encodes a key/value map into one fixed-width binary record, writing each value into the byte range the schema assigns its field.

Two things to know:

• The returned reference points at a thread-local buffer reused by the next call on the same thread. Send it or copy it before calling again. Never hold it across iterations.
• Unmatched input is dropped silently. A key absent from the schema, or a field whose byte range falls outside the record, is skipped without error. Returns an empty buffer when no schema is loaded.

append_record_to_arrow / flush_arrow_batch

bool append_record_to_arrow(const std::map<std::string, std::string>& kv_pairs);
std::shared_ptr<arrow::RecordBatch> flush_arrow_batch();

Not implemented. See Not implemented.

Batch ID generation

BatchID generate_batch_id_string(const std::string& str);
BatchID generate_batch_id_number(uint64_t number);
BatchID generate_batch_id_time(int64_t start_time, int64_t end_time,
                               int interval_override = 0);

All three produce a 64-byte ID, zero-filled and right-aligned so IDs sort and compare consistently.

generate_batch_id_time takes its interval from the schema's time index interval unless you pass interval_override; an interval of zero is treated as one second. Because the start floors and the end ceilings, the window covers at least the range you asked for and aligns to index boundaries.

Sending

bool stream_insert(const BatchID&, const std::vector<uint8_t>& record,
                   bool broadcast = false);
bool stream_insert(const BatchID&, const std::vector<std::vector<uint8_t>>& records,
                   bool broadcast = false);
bool stream_insert(const BatchID&, std::shared_ptr<arrow::RecordBatch> batch,
                   bool broadcast = false);
bool send_eob(const BatchID& batch_id, bool broadcast = false);
bool send_bbn(const BatchID& batch_id, bool broadcast = false);

stream_insert submits records to the data plane under a batch ID. The three overloads take a single encoded record, a vector of them, or an Arrow RecordBatch — the Arrow overload is the implemented path for columnar data.

send_eob closes a batch. Until the end-of-batch marker arrives, the storage layer treats the batch as still open. send_bbn sends a Bad Batch Notice instead, telling the storage layer the batch is corrupt or incomplete and should be discarded.

All five return false immediately if no SCC targets are set, and stream_insert also returns false for an empty record.

broadcast changes the destination. Left false, everything goes to the first target only. Set true, it goes to every target.

Metrics caveat: records_ingested and bytes_ingested are incremented only on the non-broadcast path. Records sent with broadcast = true are delivered but not counted, so get_metrics() will under-report if you broadcast.

Observability

std::string get_health_status() const;   // JSON
std::string get_metrics() const;         // Prometheus text format

get_health_status() returns {"status": "ok"|"initializing", "timestamp": …}, where initializing means the engine has not finished booting. get_metrics() returns records_ingested and bytes_ingested as Prometheus counters.

Both return strings. The SDK serves nothing. Publish them through whatever endpoints your own process already exposes.

FermiHDIQueryClient

The read path: dispatch queries, receive result batches over FHDWP, hand back Arrow.

Construction and teardown

explicit FermiHDIQueryClient(const std::string& proxy_url = "",
                             const std::string& data_plane_interface = "",
                             const std::string& control_plane_interface = "");
~FermiHDIQueryClient();
void stop();

An empty proxy_url selects direct SCC mode, where you supply targets yourself. The client opens a listener on port 6987 for returning result batches; inbound traffic on that port must reach this host or queries will hang.

register_with_dms

RegistrationResult register_with_dms(const std::string& dms_url,
                                     const std::string& node_name,
                                     const std::string& override_key_pem = "");

Only required when querying SCCs directly. Unlike the ingest client's version, this returns a full RegistrationResult — certificates, dataset schema and cluster topology together — so one call configures the client.

add_control_header / set_scc_targets / load_dataset_schema

As on FermiHDIIngestClient.

generate_batch_ids_time

std::vector<BatchID> generate_batch_ids_time(int64_t start_time, int64_t end_time,
                                             int interval_override = 0);

Expands a time range into one batch ID per interval — the plural counterpart to the ingest client's single-window generator. Use it to name the batches a query should read.

The range is generous at both ends: the start floors to an interval boundary and the end rounds up, then the loop runs inclusively to that rounded end. The list therefore covers at least the range you asked for and typically extends one interval past it. Filter afterwards if you need an exact window.

generate_hd_query

std::string generate_hd_query(const HDQuery& query);

Serialises an HDQuery into the JSON wire form the query methods accept. Building the struct and calling this is preferable to assembling JSON by hand, since it applies the QoS clamping for you.

Batch accounting

std::vector<BatchID> get_found_batch_ids() const;
std::vector<BatchID> get_missing_batch_ids() const;
bool had_errors() const;
std::vector<BatchError> get_batch_errors() const;

had_errors() is the only signal that a successful result is incomplete. A query whose batches were bad-batch-noticed still returns a table; it just silently lacks those records. Check it after every query. get_batch_errors() gives one entry per failed batch, not one per notice. Both reset at the start of the next query, or at prepare_query().

Executing

void execute_stream(const std::string& sql_or_json,
                    std::function<void(std::shared_ptr<arrow::RecordBatch>)> on_batch);
std::shared_ptr<arrow::Table> execute_dataset(const std::string& sql_or_json);

execute_dataset blocks until every batch has arrived and returns one Arrow Table.

execute_stream is not incremental despite its name. It dispatches the query, blocks until the result is complete, and only then walks the accumulated batches and calls on_batch once per batch. The whole result is held in memory before your first callback fires, so it saves no memory over execute_dataset() — it only changes the shape of the hand-off, from one Table to a sequence of RecordBatches. It also dispatches to the first SCC target only.

execute_dataset returns nullptr when the schema could not be loaded and the engine therefore never booted, and an empty table when the query timeout fired.

Analytics

std::shared_ptr<arrow::Table> execute_to_sql(const std::string& fermihdi_sql,
                                             const std::string& duckdb_sql,
                                             const std::string& native_args = "{}");
std::shared_ptr<arrow::Table> execute_vector_search(const std::string& sql,
                                             const std::string& vector_column,
                                             const std::vector<float>& target,
                                             size_t top_k,
                                             const std::string& native_args = "{}");
void execute_graph(const std::string& sql, const std::string& src_col,
                   const std::string& dst_col, const std::string& native_args = "{}");
void set_local_aggregations(const std::string& agg_config);

execute_to_sql runs the HD query, decodes the fixed-width records into a typed columnar table, then applies duckdb_sql to it locally. Passing an empty duckdb_sql returns the raw dataset unchanged, and if decoding fails it falls back to returning the raw data rather than failing the call. Returns nullptr only when the underlying query returned nothing.

apply_sql is the second half of execute_to_sql on its own: the SQL over a raw table already in hand. execute_sql_on_empty runs the SQL over a dataset of zero records typed by the loaded schema -- the row an aggregate produces over nothing -- for a caller whose dispatch came back with no batches to read and who therefore never ran the data plane. Both decode an empty table to the schema's typed columns first, so SELECT count(*) answers 0 rather than nothing.

The SDK does not translate values: a filter's value must already be in the column's own form. An ipv4 column takes the uint32 the address is and an ipv6 column the uint128; turning '10.0.0.1' into 167772161 is the presentation layer's job, and the storage nodes and DuckDB only ever see the integer. A filter whose value cannot be encoded is not pushed, and a sub-query left with no encodable filter means no pushdown for the whole query -- never an empty sub-query, which the storage node refuses and answers with no records and no error.

When the SQL itself fails -- a text literal against an integer column, say -- execute_to_sql returns nullptr and logs the DuckDB error; it does not fall back to the raw records, which are not an answer to the statement asked.

execute_vector_search runs the query, builds a dense vector index over the result with cosine distance, and returns the top_k nearest rows to target. Returns nullptr if the underlying query produced no table.

execute_graph and set_local_aggregations are not implemented.

Query lifecycle control

void prepare_query(const std::string& query_id);
void set_expected_batches(const std::vector<std::string>& batch_ids);
void set_filter_verdict(const std::vector<std::string>& would_exclude_batches,
                        const std::string& bloomrf_mode);
bool set_filter_verdict_json(const std::string& verdict_json);
void set_query_timeout_ms(uint64_t ms);
void cancel_query(uint32_t query_id);

prepare_query() resets query state and arms the listener before you dispatch, so end-of-batch markers arriving during dispatch are captured rather than discarded. set_expected_batches() supplies the batch list after the dispatch returns it, and completes the query immediately if everything already arrived. Together they remove the race between dispatch and the first marker; skip them and the SDK falls back to counting batches from the query payload.

set_filter_verdict() belongs to the same self-dispatch path. The SDK records the proxy's would_exclude_batches and bloomrf_mode when submit_query_to_proxy() places the query, and compares that list against the batches that actually deliver records when the query completes. A caller that dispatches by its own route has to hand the verdict over here, or the comparison has nothing to work with and never runs. set_filter_verdict_json() takes the result event's JSON as received. prepare_query() clears the previous query's verdict.

set_query_timeout_ms() caps how long a query blocks waiting for markers. The default of 0 means no timeout at all — a Storage Node that never reports will block the call indefinitely. On expiry the SDK stops waiting, sends POST /v1/query/stop to every known SCC target, and returns an empty table.

cancel_query() aborts an in-flight query by ID and is safe to call from any thread, which makes it the way to interrupt a blocking execute_dataset(). It releases the blocked call, then sends the same stop.

The stop no longer reaches the cluster. The SCC retired POST /v1/query/stop on 2026-08-09 and answers 410 Gone, naming the gRPC QueryDispatchService/CancelQuery RPC as its replacement. Both the timeout and cancel_query() still do the half that is local — your call returns, and returns promptly — but the Storage Nodes are never told, so they keep reading for that query until it finishes on its own. Cancelling frees your thread, not the cluster.

Observability

std::string get_health_status() const;
std::string get_metrics() const;

As on FermiHDIIngestClient: strings for your process to publish, not endpoints the SDK serves.

Not implemented

Declared in the public header, compile and link, but have no behaviour:

For columnar ingest today, build an arrow::RecordBatch with the Arrow API and pass it to the stream_insert overload that accepts one — that path is implemented.