Skip to content

FermiHDI Software Development Kit (SDK)

The hd_sdk repository is the unified, high-performance gateway to the FermiHDI platform. It abstracts the HD engine's kernel-bypass networking boundaries and the exact Form 1 & Form 2 TCP Protocol sequences into a polished set of developer primitives.

This single repository acts as a true Multi-Language Monorepo. We maintain a singular Zero-Copy C++ Engine natively wrapped in Python (pybind11), Go (cgo), Rust (cxx), Node.js (napi), R (Rcpp), Julia (CxxWrap), Scala/Java (JNA), and Mojo (C-FFI).

Key Features

  1. Zero-Copy Memory: Inherently relies on Apache Arrow IPC strings and DuckDB relations, removing memory copying bottlenecks inside your application loops.
  2. Dynamic Service Discovery: Built to implicitly navigate FermiHDI network topologies via the DMS and Proxy endpoints, securing X.509 sessions dynamically via Day-0 mTLS bootstrapping.
  3. Analytical Bridge: Native integration with DuckDB for complex SQL post-processing and USearch for high-density HNSW vector similarity searches.
  4. Cloud-Native Observability: Standardized OpenTelemetry JSON logging (v1.1.0), Prometheus metrics, and high-precision health reporting.
  5. 12-Factor Configuration: Externalized configuration with strict precedence and secure secrets injection (_FILE pattern).

Quick Start (Python)

pip install fermihdi-sdk

import fermihdi_sdk
import pyarrow as pa

# 1. Instantiate the Form 1 Ingester
ingester = fermihdi_sdk.IngestClient()
ingester.discover_scc_targets("https://proxy.fermihdi.io:6988")

# 2. Fire Arrow bytes purely over TCP (Zero TLS Overhead via mTLS) to the SCC Cluster 
df = pa.Table.from_arrays([[1...10000000]], names=["event_id"])
ingester.stream_insert(batch=df, broadcast=True)

# 3. Bad Batch Notice (BBN) for error signaling
# ingester.send_bbn(batch_id)

Available Sub-Engines

  1. IngestClient: Exclusively designed for pushing data into the cluster (Write Path) instantly using absolute Zero-Copy bytes over libfermihdi_transport. Supports EOB (End of Batch) and BBN (Bad Batch Notice) signaling.
  2. QueryClient: Combines standard SQL Dataframes inherently linked to dynamic embedded DuckDB engines, Hardware-Accelerated Graph processing (cuGraph/OpenMP), and USearch HNSW Vectors (cuVS)!

Documentation

The SDK is a client library: it calls endpoints and moves data over FHDWP, and serves none of its own. There is no OpenAPI specification for it, and the docs/openapi.yaml that used to be listed here described endpoints that did not exist. What the SDK calls is documented by the services that answer it — the DMS southbound and Proxy APIs.

Hardware Compatibility: AVX-512 and Windows Hyper-V

To address known SIGILL (Illegal Instruction) crashes on AMD Zen 4 and modern processors running within Windows 11 Hyper-V / Docker Desktop architectures:

The Issue: Microsoft Hyper-V has a long-standing kernel virtualization bug where it notoriously fails to context-switch AVX-512 (ZMM) registers correctly on certain modern processors. However, the hypervisor still falsely reports AVX-512 support via its CPUID broadcast to the guest containers. Standard C++ compilers, pre-compiled wheels, and dynamic analytical dispatching engines (like OpenMP, SYCL, DuckDB, PyArrow) will trust this CPUID flag, immediately attempt to vectorize high-speed matrices, and flawlessly crash their own containers via illegal instruction traps (Exit Code: -4).

The Solution: Use the built-in DISABLE_AVX512 toggle to structurally override vectorization: 

export DISABLE_AVX512=1
# Or explicitly via CMake
cmake -B build -DDISABLE_AVX512=ON
Activation safely guarantees stability on Docker Desktop while retaining full AVX2 functionality.