Cognitora

Cognitora is an open-source orchestration layer for distributed large language model inference, written in Rust. Rather than shipping a single inference engine, it joins the engines you already operate — vLLM, SGLang, llama.cpp, Apple MLX, a TRT-LLM spawn driver and its own preview cgn-infer engine — into one cluster behind a single OpenAI-compatible gateway. You can deploy it instead of an orchestrator such as NVIDIA Dynamo, and the project positions itself as supporting more engines, running on bare metal without Kubernetes, factoring power consumption into scheduling and avoiding a Python control plane.

Its core innovation is KV-aware routing. The router scores nodes by the longest-prefix overlap of sequence-chained BLAKE3 digests, using cache claims that agents report in heartbeats. In real llama.cpp runs the router reportedly added 0.9–2.0 ms at p50. Beyond routing, the platform splits prefill from decode across separate pools (handing KV cache off through NixlConnector), maintains a multi-tier RAM/SSD KV cache with cross-node QUIC fetch, and offers pluggable offload backends including nixl, lmcache, hicache and kvbm.

The stack is modular: cgn-router for routing, cgn-agent to supervise engines on each node, cgn-kvcached for tiered caching, cgn-metrics for Prometheus plus Redfish/NVML power telemetry, cgn-ctl as an admin CLI, an optional Kubernetes operator, and a preview native engine. Energy-aware scheduling drains hot nodes, a closed-loop autoscaler reacts to power readings, and gossip-based discovery removes the need for etcd. Security options include optional mTLS, SHA-256 checksummed releases and OIDC or scoped API-key auth. The project ships six static binaries for systemd installs, Helm charts for Kubernetes, and a single-file HTML cluster dashboard that visualises throughput, latency percentiles, KV utilisation and joules per token without Grafana.