FermiHDI SDK — Scala

BETA. Compiles on every change. JDK 17 only — see below. Narrow surface; expect to extend it.

Scala binds the C ABI of the shared library through JNA, which means the JVM loads libfermihdi_client.so at runtime and the SDK's threads run beside the JVM's own. The SDK overview sets out the model those threads implement; read §1, the control plane / data plane split before anything here.

Modes

JNA reaches the C ABI's query client. There is no registration call and no ingest client behind it yet, so from Scala the SDK is used in proxy mode, submitting through the Query Proxy with records arriving over the data plane; SCC targets can be set explicitly, but the JVM process cannot register as an HD_CLIENT node or ingest. See the SDK overview for the two modes and the bindings that support each.

Build

JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 sbt compile

JDK 17 is a hard requirement. scalaVersion is pinned to 3.3.0, which cannot read the class files of JDK 21 or newer — it fails while loading java.base itself:

error while loading AccessFlag, class file .../AccessFlag.class is broken
bad constant pool index: 0

That is the compiler, not this binding. Raise scalaVersion if you need a newer JDK.

The JVM dlopens the library, so you also need the static TLS surplus:

GLIBC_TUNABLES=glibc.rtld.optional_static_tls=1048576 sbt run

API

trait SDKCore {
  def discoverSCCTargets(proxyUrl: String): Boolean
  def executeDataset(sql: String): Boolean
}

class NativeQueryClient(proxyUrl: String) {      // over the C ABI
  def prepareQuery(queryId: String): Unit          // arm the listener FIRST
  def setFilterVerdict(verdictJson: String): Boolean
  def filterFalseNegatives(): String              // JSON: bloomrf_mode, false_negatives
  def close(): Unit
}

setFilterVerdict is for a query you dispatched yourself: pass the proxy's result event JSON verbatim so the SDK knows what the filter service would have excluded. Without it the false-negative check at the end of the query never runs. filterFalseNegatives must report an empty list.

New calls are JNA declarations against the C ABI — flat, nothrow and stable, and a far better target than the C++ symbols.

Example

import com.sun.jna.Native

val sdk = Native.load("fermihdi_client", classOf[SDKCore])
if (!sdk.discoverSCCTargets("https://proxy.internal:443"))
  sys.error("no storage cluster controllers discovered")

Use cases

Spark or Flink jobs that need HD as a source without a Python hop. Pull a window into Arrow and hand it to the JVM pipeline.

Extending the binding. Map fhd_* from fermihdi_c_api.h; the buffer- filling *_into variants avoid JNA memory-ownership questions entirely.

Gotchas

• JDK 17. Anything newer fails to compile Scala 3.3.0.
• You own the ending — relay complete or cancel.
• Query IDs must be distinct across all clients; seed at random.