Kinference

Kinference (pronounced kay-inference) is an open-source inference control plane built by Buoyant, the company behind the Linkerd service mesh, and announced on the vLLM blog on 2026-11-04. The code lives at GitHub Kinference-AI/kinference.

It aims to be the missing layer for building an inference platform on vLLM and Kubernetes: point OpenAI- or Anthropic-compatible clients at Kinference's HTTP endpoint and it handles token rate limiting, per-tenant usage and cost tracking, intelligent load balancing and inference policies, while vLLM does the actual inference.

What problem it solves

Teams self-hosting open-weight models juggle many models, heterogeneous GPU pools and multiple tenants. Standard Kubernetes Service load balancing doesn't understand LLMs, wasting KV cache and making cost control hard.

Core features

• KV-cache-aware load balancing: routes each request to the vLLM worker that will do the least work, using GPU queue sizes, historical performance and modeled KV-cache contents.
• Inference policies: e.g. send simple requests to a cheaper model only when that is less work given the cache state.
• Governance: token rate limiting, tenant usage and cost tracking, metrics and SLOs.

Official benchmarks

• vs. default Kubernetes Service balancing: 95% lower median TTFT and 45% lower median end-to-end latency on one workload.
• Capacity test: 7.4× goodput on shared-prefix and about 3.5× on multi-turn workloads.

Limitations

• Brand-new project; ecosystem and production references are still growing.
• Requires Kubernetes Gateway API CRDs and vLLM; not for non-K8s setups.

Who it's for

Platform engineering and MLOps teams running multi-model, multi-tenant vLLM inference on Kubernetes.

Getting started

1. Read the vLLM blog announcement and GitHub Kinference-AI/kinference.
2. Follow the four steps: install Gateway API CRDs → install the Kinference gateway → install Kinference → verify it works.
3. Point your existing OpenAI/Anthropic-compatible clients' base URL at the Kinference endpoint.
4. Configure tenants, rate limits and routing policies, and wire metrics into monitoring to watch TTFT and goodput SLOs.