One open-source platform to schedule, serve, benchmark and observe LLMs on NVIDIA and AMD GPUs.
Open Inference Fabric (OIF) assembles open-source AI infrastructure into a suite you can install on GPU servers with one command:
| Layer | Built on |
|---|---|
| Orchestration | NVIDIA Run:ai (open source) with the KAI Scheduler: projects, quotas, node pools |
| Serving | NVIDIA Dynamo with vLLM (SGLang and TensorRT-LLM later), one OpenAI-compatible endpoint; AMD GPUs through a ROCm build |
| Performance | NVIDIA AIPerf benchmark profiles and comparison reports |
| Observability | Prometheus and Grafana with DCGM and AMD device metrics exporters, normalised into one vendor-neutral schema |
| Platform | Upstream Kubernetes (kubeadm), NVIDIA GPU Operator, AMD GPU Operator |
Status: v0.1 in development. Multi-node HA and dual-vendor orchestration are advanced; serving and gateway are next. Start with the install guides; progress is in TASKS.md.
Teams with both NVIDIA and AMD GPUs run separate stacks per vendor: different serving endpoints, different dashboards, no shared quotas, and benchmarks that cannot be compared. OIF puts both vendors behind one scheduler, one API, one benchmark tool and one set of dashboards — with a clean, reversible installer.
./oif check # read-only host and GPU report
cp .env.example .env && $EDITOR .env # addresses, tokens
sudo ./oif install # questions first, then unattended
sudo ./oif status | plan # progress; preview of uninstall
sudo ./oif uninstall # back to the OS default
sudo ./oif bench --model qwen3-0.6b # (Project 2) AIPerf sweep on each GPU, HTML report| Document | Purpose |
|---|---|
| Project brief | One page for partners, sponsors and new contributors |
| PRD | What we are building and why; goals and success criteria |
| SDLC | How we build, test, document and release (process for humans and agents) |
| Architecture | Layers, flows, telemetry schema, repository layout |
| Design | CLI, dashboard and report conventions |
| Rules | How code is written and changed |
| Tasks | What we build next, in order |
| Decisions | Architecture Decision Records (ADRs) |
| Memory | Current project state |
| Test plan | What "working" means |
| Install guides | Step-by-step install and deployment |
| Upstream changes | Every patch and workaround, prepared for the Run:ai and KAI Scheduler projects |
| Design sketches | Vendor-agnostic job model, lab acceptance checks |
| Presentations | Feature and product decks |
| Security | Security requirements |
| Research | Sources behind the decisions |
Developed and tested on one server: AMD EPYC 9684X, 751 GiB RAM, NVIDIA RTX A6000 and AMD Radeon PRO W7900, Ubuntu 26.04. Multi-node work uses the hybrid-gpu-lab.
Testers with NVIDIA, AMD or mixed GPU servers, reviewers for the upstream Run:ai and KAI work (docs/UPSTREAM.md), and contributors are all welcome: see CONTRIBUTING.md and the Code of Conduct. Report security problems privately (docs/SECURITY.md).
Apache-2.0. NVIDIA, Run:ai, Dynamo, AMD, ROCm and Radeon are trademarks of their owners; this project is not affiliated with or endorsed by them.