docs.fermihdi.io — js.md

/ › hd › 2.0.4 › md › sdk

Download js.mdView raw

FermiHDI SDK — JavaScript / Node.js

BETA. Builds and links on every change. Not yet exercised against a live cluster as heavily as the Go binding.

An N-API addon over the C++ SDK. Read the SDK overview first, especially §1, the control plane / data plane split — records never come back through the connection you submit on.

Build

export FERMIHDI_SDK_ROOT=/opt/fermihdi-sdk
npm install
npx node-gyp rebuild

binding.gyp resolves headers and libraries through FERMIHDI_SDK_ROOT (default /opt/fermihdi-sdk) and sets two flags that are not optional:

It also names the full library list, because -lfermihdi_client alone does not pull in the HD engine runtime's dependencies.

You will need the TLS tunable

Node dlopens the addon, and the SDK carries the HD engine runtime's initial-exec TLS:

GLIBC_TUNABLES=glibc.rtld.optional_static_tls=1048576 node app.js

Otherwise require() fails with cannot allocate memory in static TLS block — see §5.3.

API

const { QueryClient } = require('./build/Release/fermihdi_sdk');

const client = new QueryClient(proxyUrl);

client.register_with_dms(dmsUrl, nodeName)   // -> { success, schema }
client.load_dataset_schema(schemaJson)
client.generate_batch_ids(from, to)          // -> [hex string]
client.generate_batch_ids_time(fromSec, toSec)
client.generate_hd_query(hdQueryObject)      // -> query body JSON, "" if refused
client.prepare_query(queryId)                // arm the listener FIRST
client.set_expected_batches(batchIds)        // after the control plane answers
client.set_filter_verdict(wouldExclude, mode) // the proxy's would_exclude_batches, bloomrf_mode
client.filter_false_negatives()              // -> { bloomrf_mode, false_negatives }
client.execute_to_ipc(queryJson, duckdbSql)  // -> Buffer of Arrow IPC
client.execute_empty_to_ipc(duckdbSql)       // the SQL over zero records, when nothing was dispatched
client.execute_dataset_to_ipc(queryJson)     // the raw record table, untouched, as Arrow IPC
client.apply_sql_to_ipc(ipcBuffer, duckdbSql) // SQL over a table you hold, in and out as Arrow IPC
client.stop()

Worked example

const { QueryClient } = require('./build/Release/fermihdi_sdk');
const crypto = require('crypto');

const client = new QueryClient('https://proxy.internal:443');
client.load_dataset_schema(schemaJson);

const batchIds = client.generate_batch_ids_time(1788500000, 1788503600);
const queryId  = crypto.randomInt(0, 2 ** 32);   // distinct per query

const body = client.generate_hd_query({
  query_id: queryId,
  batch_ids: batchIds,
  targets: [{ ip: advertisedIp, port: 6987 }],
  // An address is its integer: 10.0.0.1 is 167772161. Convert before the SDK sees it.
  sub_queries: [{ filters: [{ field_name: 'ip_src', value: '167772161', op: 0 }] }],
});
if (!body) throw new Error('open query refused without acknowledgement');

// 1. arm the listener before anything can answer
client.prepare_query(String(queryId));

// 2. control plane — your own HTTP call to the proxy
const found = await dispatchToProxy(proxyUrl, body);

// 3. wait for exactly what exists, and say what the filter would have skipped
client.set_expected_batches(found);
client.set_filter_verdict(wouldExclude, bloomrfMode);   // from the same result event

// 4. data plane — blocks until every expected batch has arrived
const ipc = client.execute_to_ipc(body, 'SELECT ip_src, bytes FROM ipflows LIMIT 10');

// 5. report the ending
await reportQueryEnd(proxyUrl, queryId, 'complete');

client.stop();

execute_to_ipc returns a Buffer of Arrow IPC bytes; decode it with apache-arrow:

const { tableFromIPC } = require('apache-arrow');
const table = tableFromIPC(ipc);

Use cases

A Node API in front of HD. Hold one client for the process lifetime, mint a distinct query ID per request, and relay endings. Note that execute_to_ipc blocks the event loop — run it in a worker thread if the process serves other traffic.

A CLI or ETL step. Blocking is fine; keep it simple.

Gotchas

Download js.mdView raw