Sandbox Data Management System (DMS) Operators & Admins Manual

The ./tests/sandbox_dms directory provides a self-contained, containerized FermiHDI Data Management System (DMS) tailored for local development, testing, and benchmarking. It deploys the entire ecosystem securely, mimicking a production-like Data Plane and Control Plane without requiring external infrastructure.

The sandbox is also the reference deployment the installer is built from and checked against: compose.yml is every service with its full environment, and generate_certs_and_license.sh is the clearest statement of the PKI an install needs. A deployment you can read: the sandbox reads the two side by side, including where the sandbox is deliberately not production.

Architecture

The sandbox provisions the following components via Docker Compose:

• DMS-Core: The central control plane component responsible for orchestrating HD instances.
• DMS-CA: The internal Certificate Authority responsible for provisioning mutual TLS (mTLS) identities.
• DMS-WebUX: The user interface for interacting with the cluster.
• HD Proxy: The high-performance entry point evaluating PBAC policies via OPA.
• Storage Cluster Controllers (SCC): Nodes managing horizontal data synchronization across Storage Nodes.
• Storage Nodes (SN): The data layer managing NVMe/file-backed Data Plane retrieval.

Networking Rules

The sandbox strictly enforces FermiHDI network segmentation:

• 6985: Internal DMS Plane (DMS-Core <-> DMS-CA, WebUX)
• 6986: Control Plane (HTTPS mTLS) for queries and orchestration.
• 6987: Data Plane (FHDWP) for unidirectional, high-speed query response.
• 6988: Auxiliary Network (NORM Multicast)
• 6989: DNS CA Network
• 8080: Observability (Prometheus Metrics & Health)

All containers are completely isolated on an internal bridge network (isolated_network), avoiding host port conflicts unless explicitly mapped.

Operating the Sandbox

Bootstrapping

To start the Sandbox in an automated manner:

cd ./tests/sandbox_dms
./start_sandbox.sh

The script performs the following actions:

1. Cleans up any existing sandbox state to prevent cross-contamination.
2. Regenerates cryptographic key material (Root CA, Intermediate CA).
3. Evaluates CPU architecture and builds the relevant Docker Compose topology.
4. Executes docker compose up -d.

Storage Node count

--num-sns N (1, 2 or 3) selects how many Storage Nodes run. When you do not pass it, the count recorded in sandbox_hw_config.json — what the host was actually provisioned for by configure_sandbox.py — is used, and the script warns that it differs from the nominal default of 3. An explicit --num-sns always wins, and --standalone forces 1.

This matters because three separate consumers must agree on the number: compose starts that many SN containers, configure_sandbox.py writes matching registration records into nodes.jsonl, and rebalance_sandbox.py trims the rest. If they disagree, the extra SNs come up with no provisioning record and loop every 30 seconds on:

DMS is in Setup Mode (503 Service Unavailable). Awaiting License Context.

That is how DMS-Core answers /v1/register for a node it has never heard of, and it never resolves. The symptom is easy to miss: those containers still report their control plane healthy, and the script still prints "Sandbox is fully launched!" — only the storage table at the end, listing fewer drives than SNs, gives it away.

Version note. Before 2.0.3 the script's own default of 3 silently beat the provisioned value, so a host configured for one SN would start three and wedge two of them. If you are on an earlier build, pass --num-sns explicitly to match sandbox_hw_config.json, or re-run configure_sandbox.py.

Customizing Configuration

Two variables exist for watching a certificate renewal happen in a running lab rather than waiting for one: FERMIHDI_CERT_RENEW_DAYS widens the renewal window (a value larger than the 30-day certificate lifetime renews at the first check) and FERMIHDI_CERT_RENEW_INTERVAL_SECONDS shortens the daily check. Export both before the launcher; every node and the Proxy read them, and each renewal is logged with the certificate's new expiry. Neither belongs in a production deployment.

Administrators can modify the deployment via environment overrides.

• CPU Set Pinning: Edit the docker-compose.yml to define explicit CPU affinities for high-throughput testing:

  cpuset: "0-3" # Pin SCC to first 4 logical cores
  

• Dataset Schema Injection: The proxy schema can be pre-loaded by modifying the environment variable FERMIHDI_DATASET_SCHEMA_FILE.

Diagnostics & Troubleshooting

• Validating Logs:

  docker compose -f ./tests/sandbox_dms/docker-compose.yml logs -f dms-core
  

• Checking Data Plane Connectivity: Ensure port 6987 is accessible natively if querying from outside the isolated bridge network.
• Verifying Observability: Navigate to http://localhost:8080/metrics on any given node container.

Shutdown & Cleanup

Always ensure proper teardown to release isolated networks:

cd ./tests/sandbox_dms
docker compose down -v --remove-orphans

Note: Never issue blanket commands like docker rm -f $(docker ps -aq).