Skip to content

Python SDK Integration Guide

Overview

The FermiHDI Python SDK is a high-performance native bridge to the C++ fermihdi_client. It leverages pybind11 to expose zero-copy Apache Arrow matrices, hardware-accelerated bitmasking, and embedded analytics (DuckDB/USearch) directly into Python.

Authentication & Network Modes

The SDK operates in two distinct modes for queries:

  1. Proxy Mode (Default):
  2. Connects to the Proxy URL over Standard TLS (Port 443).
  3. Automatically retrieves the Dataset Schema dynamically from the Proxy /v1/hd/schema endpoint.
  4. Requires external Forward Auth (Identity Provider) handling.

  5. The Proxy manages discovery and SCC routing.

  6. Direct Mode (mTLS):

  7. Connects to the SCCs over mTLS (Port 6987) for the Control Plane (Query submission).
  8. Requires calling register_with_dms() once to bootstrap the topology and signed mTLS certificates.
  9. All record streaming occurs on the Production Network via the unique FermiHDI HD Wire Protocol Form 2.

Setup

Install via pip (ensure the C++ shared library is in your system path), then import the wrapper: 

pip install .

The SDK provides a pure Python wrapper module fermihdi which safely wraps the native Pybind11 _fermihdi_sdk_cpp library and automatically detaches background hardware threads via atexit hooks. Additionally, you should prefer the with Context Manager syntax for absolute safety.

Query Client (Form 2 Reads)

Basic Usage: Proxy Mode

import fermihdi as fsdk
import json

# Initialize for Proxy Mode safely inside a Context Manager block
with fsdk.QueryClient("https://proxy.fermihdi.io") as client:

    # 2. (Optional) Load the Dataset Schema manually
    # If omitted, this will instantly fetch via the proxy /v1/hd/schema endpoint in the background!
    # client.load_dataset_schema(schema_json)

    # 3. Generate Batch IDs for the temporal window
    batches = client.generate_batch_ids_time(1700000000, 1700003600)

    # 4. Construct high-level Filters
    # These are automatically translated into 64-byte bitmasks by the C++ engine.
    hq = fsdk.HDQuery()
    hq.query_id = 1001
    hq.batch_ids = batches

    sq = fsdk.SubQuery()
    sq.filters = [
        fsdk.Filter("price", "5000", fsdk.FilterOp.GREATER),
        fsdk.Filter("category", "auth", fsdk.FilterOp.EQUALITY)
    ]
    hq.sub_queries = [sq]

    # 5. Generate the JSON HD Query
    query_json = client.generate_hd_query(hq)

    # 6. Execute and Apply DuckDB sub-query
    # Blocks until all Storage Nodes stream thier Form 2 packets.
    table = client.execute_to_sql(query_json, "SELECT * FROM dataset WHERE price > 10000")

    if table:
        print(f"Records extracted: {table.num_rows}")

# Context Manager automatically halts the underlying engine reactors deterministically!

Direct Mode with DMS Registration

with fsdk.QueryClient() as client: # No proxy URL
    # Bootstrap mTLS and Topology
    reg = client.register_with_dms("https://dms.fermihdi.io", "my_analytics_node")

    if reg.success:
        print("Direct mTLS Mode Enabled.")
        # In Direct Mode, the SDK communicates directly with the SCCs returned in reg.scc_targets

Ingest Client (Form 1 Writes)

with fsdk.IngestClient() as ingester:
    ingester.discover_scc_targets("https://proxy.fermihdi.io")

    # Map record to correctly aligned binary bytes based on Schema
    binary_record = ingester.convert_record_to_binary({"price": "1234", "symbol": "BTC"})

    # Transmit Form 1 packet to SCCs on Production Network
    ingester.stream_insert("batch_id_hex_string", binary_record)

Shared Clients Pipeline (Advanced)

If context-managers cannot be used (such as inside global multi-process architectures), use the shared client pipeline:

ingest_client = fsdk.get_shared_ingest_client("10.0.0.1", 6986)
b_id = ingest_client.generate_batch_id_string("my_batch")
ingest_client.stream_insert(b_id, raw_bytes, is_eof=True)

# You can manually clear them at any time, or rely on `atexit` hook during teardown.
fsdk.clear_shared_clients()