Skip to content

FermiHDI SDK Architecture Map

1. Architectural Design

The Core Monorepo Concept

FermiHDI targets ultra-low latencies. If every single language wrapper were to natively implement the Form 1 wire protocol locally in Go, Python, and Javascript, it would heavily limit the system's ability to maintain a strictly controlled engine memory pool. It would cause exponential tech debt over separate repositories whenever a single Wire format changes.

Instead, the hd_sdk is an inverted dependency graph: 1. The HD engine's transport layer: handles the networking bytes, the TCP sequences, and the event loops that map onto kernel-bypass interfaces. 2. fermihdi_client (C++): Maps network functions into explicit execute_stream, execute_graph, and execute_vector_search boundaries, wrapping types transparently and safely. 3. Foreign Language Wrappers: Pass native pointers and strings smoothly through explicit FFI interfaces directly into fermihdi_client.

The Networking Flow

When building applications: * The Control Network (TLS/mTLS): Is contacted once leveraging register_with_dms to efficiently retrieve the .SCRoutingTable, identifying which specific Database Node IPs the client should target automatically. * The Production Network (TCP/NORM): The actual heavy lifting operations (pulling billions of RecordBatches sequentially) bypass TLS encryption, explicitly mapping native Arrow frames dynamically over the engine's kernel-bypass channels.

2. Using the C++ Backend natively

If you are developing a C++ application that requires maximum native integration:

#include <fermihdi/sdk/FermiHDISDK.hpp>
#include <arrow/api.h>

using namespace fermihdi::sdk;

int main() {
    FermiHDIQueryClient db("https://proxy.fermihdi.io");

    // Explicitly Register against Control Plane natively
    db.register_with_dms("https://dms.fermihdi.io", "my_node_1");

    // Evaluate Data using USearch Native AVX-512 integration directly
    std::vector<float> embed = {1.0, 0.5, 0.4};
    auto similar = db.execute_vector_search("SELECT text FROM embeddings", "vector", embed, 5);

    return 0;
}

3. Hardware Acceleration & Embedded Engines

The fermihdi_client natively encapsulates advanced analytics directly into the returning Arrow IPC buffers: - DuckDB: Instead of looping across arrays via generic strings, the Dataframe explicitly wraps an embedded duckdb::DuckDB db zero-copy memory pointer executing SQL aggregations instantaneously. - Graph Mathematics (OpenMP / cuGraph): Bypasses NetworkX completely. It instantiates Native PageRank/SSSP directly alongside arrow::Int64Array relationships across Nvidia CUDA cores or Multi-Core Intel SIMD architectures. - USearch (HNSW Vectors): Binds the raw vectors directly into usearch/index_dense.hpp executing Cosine distances via AVX-512 instantly on return.

4. Language Matrix Integration Status

  • Python: Available via pybind11. Used for PyArrow/Pandas seamless zero-copy integration.
  • Go: Available via cgo. Best for Kubernetes operators/microservices and concurrent Data Engineering.
  • Rust: Bound strictly using safe cxx::bridge. Ideal for building hyper-secure data endpoints.
  • Node.js: Available via napi. Directly pipes Web Gateway events (Express/Bun) to the SCC natively.
  • R (Rcpp) & Scala (JNA): Evaluates statistics directly bypassing JVM limits or R memory boundaries completely.
  • Mojo & Julia: Hooks Form 1 strings natively, allowing state-of-the-art Generative AI processing limits.