Skip to content

What's new

New features and new functionality in each release of HD, newest first. Defect corrections are in each component's release notes and are not repeated here.

2.0.5

New features

  • Chess notation fields. A chess_move field stores a move in standard algebraic notation as a 16-bit code, and a chess_fen field stores a full board position in 32 bytes. Queries show both as notation, and position filters (a piece on a square, the side to move) are evaluated on the storage nodes. See Chess notation.
  • Chess board view in Data Explore. Data Explore replays a game from its starting position and its list of moves on an interactive board, and filters games by position.
  • Array fields. A field can hold a fixed-width list of values. Filters that ask whether some element matches, or whether none does, run on the storage nodes; the filter node prunes batches for the "some element matches" form. Every query service returns arrays as native lists. See Arrays.
  • Enumerations and 8-bit text. A dataset can declare named value lists. Fields store compact codes and every query service shows and accepts the labels. ASCII text fields are stored one byte per character. See Enumerations.
  • Intra-record time compression. A time field can be stored as a small delta from another time in the same record, in a unit of your choosing, and every query service returns the full timestamp. See Time fields.
  • Opt-in cloud sharing. A single, deployment-wide switch decides whether anonymised operational telemetry is shared with FermiHDI. It is off by default and enforced at the deployment's own collector. See Data handling and opt-in.
  • Crash reporting and the crash vault. Crashes in the engine roles, DMS-Core, the Proxy and the WebUX are captured, held and symbolicated on the DMS host. They are shared only as an operator decides, and can be exported for air-gapped support.
  • Bare-Docker multi-host deployment. The installer can render plain docker run scripts for each host from the cluster file, for deployments with no orchestrator.
  • Signed images with provenance. Every image is signed, carries labels naming its source repository, commit and branch, and ships with vulnerability exploitability statements. See Security and development practices.
  • Ubuntu 26.04 image variants. Every service is published as an OCI Rock and as an Ubuntu 26.04 image, each for AVX-512 and AVX2 processors.
  • Agent Skills. Every installer tool, the sandbox, troubleshooting, hardware sizing, schema authoring, ingestion setup and the query services have an Agent Skill an AI coding assistant can load to drive them conversationally. Published at downloads.fermihdi.io/skills/. See Agent Skills.

New functionality

  • Dry-run ingestion. The ingester's --dry-run mode reads, decodes and maps records against a dataset schema exactly like a real ingest, but registers with no DMS, connects to no SCC and needs no licence — verify a data source and a schema are compatible before a real deployment sees either. See Verifying a source before it feeds a deployment.

  • Jumbo frames end to end. FERMIHDI_JUMBO enables 8500-byte data-plane frames on the ingester, the SCC and the storage nodes alike. A record wider than a standard frame's payload now travels whole in a frame of its own.

  • Closing the last batch. The ingester closes an idle batch after a timeout, set manually or derived from the time span of the files it reads. The last batch of a file drop no longer waits for the next delivery. See Closing the last batch.
  • Exact 64-bit values. Integers and nanosecond timestamps larger than 2^53 are stored exactly from every file format and from quoted JSON values.
  • Larger filters. A query whose filter exceeds 64 KB, such as a long list of OR conditions, now reaches the storage nodes.
  • Time-series windows on the batch field. Data Explore windows and buckets a time series on the dataset's batch identifier field.
  • Schema import from a share link. The WebUX imports a dataset schema from the schema generator's share link.
  • Filters that survive restarts. The filter node keeps its filters on local storage, and every node fetches its configuration again when it restarts.
  • Filter node licensing. An SCC takes an HD_BLOOMRF licence slot to run its filter in active mode, and runs it passive when no slot is available. The filter node itself needs no slot.
  • Syslog export over TLS or plain. The WebUX's syslog exporter has a transport setting (TLS, plain, or by port), and each message carries the configured facility, the record's severity, host name and service. See OTEL Collector.
  • Hardened Kubernetes defaults. The management and query services run without privilege escalation or raw-socket access and under the default system-call filter. The chart generates the internal access token when none is supplied.

2.0.4

New features

  • The filter node. An optional BloomRF service per storage cluster. In active mode it tells the SCC which batches a query cannot match, so they are never read; passive mode measures the same without acting on it. See The filter node.
  • The SQL server. A read-only SQL front door over the PostgreSQL wire protocol, Arrow Flight SQL and HTTP, for BI tools, notebooks and JDBC, ODBC and ADBC clients. See The SQL server.
  • Talos Linux. A cluster whose machines run Talos Linux is installed from the same cluster file, over the Talos API.
  • File and RAM storage. Storage nodes can run on a file or a RAM device as well as on NVMe drives.
  • Online licence activation. Licences are activated against the FermiHDI licence service, encrypted and bound to the DMS they were issued for, and capacity is enforced.
  • Installer archive. The installer and its tooling are published as a versioned download.
  • Query cancellation and open queries. A client can cancel a running query through the Proxy, and a query may omit its time range to read the whole dataset.
  • Batch-level query policy. Policy can restrict which batches a query is allowed to read.

New functionality

  • File and object ingestion. The ingester reads Parquet, ORC, Avro, Protobuf and XML from directories and S3-compatible object stores, sorting each file's records into batches by the schema's batch field. See Ingestion.
  • IPv4 and IPv6 addresses. Address fields are converted at every query service boundary, in both directions, exactly.
  • Blob fields are supported by the SDK and every query service.
  • In-place certificate renewal. Node certificates renew themselves before they expire, without a restart.
  • Separate hugepage pools. Networking uses 2 MB hugepages and storage 1 GB hugepages, each discovered at start.
  • Filter node provisioning in the WebUX, with the filter mode set per deployment and each SCC's filter state shown.
  • Query stream lifetime. The query services hold the client stream open for the life of a query, with an explicit completion signal and a liveness signal on idle connections.
  • Excluded batches reported. When the filter node excludes batches, the SCC reports them to the client, so a result is known to be complete.

2.0.2

New features

  • The installer. Discovery, configuration, validation and preparation of a whole cluster from one cluster file, deployed with Ansible on bare metal or with Helm on Kubernetes.
  • Query priority. A query carries a quality-of-service weight. Storage nodes schedule work by it with weighted fair queuing, and policy can override it.
  • Policy-based access control in the Proxy. Policy can inject filters into a query and adjust its priority.
  • Verified batches. A batch is finalised only after every storage node that received it has verified it, and each batch is recorded with its closing hash.
  • Direct NVMe storage. Storage nodes drive NVMe devices from user space, with RAID-0 across devices and support for SATA, SAS, Fibre Channel, iSCSI, RAM and NVMe over Fabrics.
  • Observability endpoints. Every engine node exposes health, recent logs and metrics, and exports metrics over OpenTelemetry.

New functionality

  • Topology-aware CPU placement. Cores are allocated by NUMA node, cache domain and physical core.
  • Faster, larger queries. Wide queries stream their batch lists, and the query relay on the storage nodes was redesigned with bounded memory and admission control.
  • Configurable chain hashing, including the XXH3 chain with a tunable block size.
  • Finer timings. Latency is reported in microseconds.