Python SDK Integration Guide

Overview

The FermiHDI Python SDK is a high-performance native bridge to the C++ fermihdi_client. It leverages pybind11 to expose zero-copy Apache Arrow matrices, hardware-accelerated bitmasking, and embedded analytics (DuckDB/USearch) directly into Python.

Authentication & Network Modes

The SDK operates in two distinct modes for queries:

Proxy Mode (Default):
- Connects to the Proxy URL over Standard TLS (Port 443).
- Automatically retrieves the Dataset Schema dynamically from the Proxy /v1/hd/schema endpoint.
- Requires external Forward Auth (Identity Provider) handling.
- The Proxy manages discovery and SCC routing.
Direct Mode (mTLS):
- Connects to the SCCs over mTLS (Port 6987) for the Control Plane (Query submission).
- Requires calling register_with_dms() once to bootstrap the topology and signed mTLS certificates.
- All record streaming occurs on the Production Network via the unique FermiHDI HD Wire Protocol Form 2.

Setup

Install via pip (ensure the C++ shared library is in your system path), then import the wrapper:

pip install .

The SDK provides a pure Python wrapper module fermihdi which safely wraps the native Pybind11 _fermihdi_sdk_cpp library and automatically detaches background hardware threads via atexit hooks. Additionally, you should prefer the with Context Manager syntax for absolute safety.

Query Client (Form 2 Reads)

Basic Usage: Proxy Mode

import fermihdi as fsdk
import json

# Initialize for Proxy Mode safely inside a Context Manager block
with fsdk.QueryClient("https://proxy.fermihdi.io") as client:
    
    # 2. (Optional) Load the Dataset Schema manually
    # If omitted, this will instantly fetch via the proxy /v1/hd/schema endpoint in the background!
    # client.load_dataset_schema(schema_json)
    
    # 3. Generate Batch IDs for the temporal window
    batches = client.generate_batch_ids_time(1700000000, 1700003600)
    
    # 4. Construct high-level Filters
    # These are automatically translated into 64-byte bitmasks by the C++ engine.
    hq = fsdk.HDQuery()
    hq.query_id = 1001
    hq.batch_ids = batches
    
    sq = fsdk.SubQuery()
    sq.filters = [
        fsdk.Filter("price", "5000", fsdk.FilterOp.GREATER),
        fsdk.Filter("category", "auth", fsdk.FilterOp.EQUALITY)
    ]
    hq.sub_queries = [sq]
    
    # 5. Generate the JSON HD Query
    query_json = client.generate_hd_query(hq)
    
    # 6. Execute and Apply DuckDB sub-query
    # Blocks until all Storage Nodes stream thier Form 2 packets.
    table = client.execute_to_sql(query_json, "SELECT * FROM dataset WHERE price > 10000")
    
    if table:
        print(f"Records extracted: {table.num_rows}")
        
# Context Manager automatically halts the underlying engine reactors deterministically!

Direct Mode with DMS Registration

with fsdk.QueryClient() as client: # No proxy URL
    # Bootstrap mTLS and Topology
    reg = client.register_with_dms("https://dms.fermihdi.io", "my_analytics_node")
    
    if reg.success:
        print("Direct mTLS Mode Enabled.")
        # In Direct Mode, the SDK communicates directly with the SCCs returned in reg.scc_targets

Ingest Client (Form 1 Writes)

with fsdk.IngestClient() as ingester:
    ingester.discover_scc_targets("https://proxy.fermihdi.io")
    
    # Map record to correctly aligned binary bytes based on Schema
    binary_record = ingester.convert_record_to_binary({"price": "1234", "symbol": "BTC"})
    
    # Transmit Form 1 packet to SCCs on Production Network
    ingester.stream_insert("batch_id_hex_string", binary_record)

Shared Clients Pipeline (Advanced)

If context-managers cannot be used (such as inside global multi-process architectures), use the shared client pipeline:

ingest_client = fsdk.get_shared_ingest_client("10.0.0.1", 6986)
b_id = ingest_client.generate_batch_id_string("my_batch")
ingest_client.stream_insert(b_id, raw_bytes, is_eof=True)

# You can manually clear them at any time, or rely on `atexit` hook during teardown.
fsdk.clear_shared_clients()

docs.fermihdi.io — python-integration.md