The AI gateway that runs where your data lives.

Datacenter throughput, as a Python service.

vLLM is built for high-throughput serving on GPU clusters and comes with a large Python dependency set. Choose it for cluster-scale serving. OneAI targets a single node you control, ships as one native binary with no Python, Node or Docker, and adds the gateway, policy and provider layers that vLLM leaves to other tools.

A local runner for individual developers.

Ollama makes running a model on a laptop easy and is built on the same llama.cpp/ggml stack. OneAI is built for putting local models in front of applications and teams: application keys with model grants and limits, a published catalogue, provider routing with fallback and budgets, guard models, PII redaction, an emergency stop, Prometheus metrics and OpenTelemetry export.

llama.cpp's own HTTP server.

An excellent raw engine with a minimal management layer, typically one model per process. OneAI wraps the same engine with continuous batching across resident models, hot swap and LRU eviction, automatic placement, prompt and response caches, encoders and speech, plus the full gateway around it.

Routing between hosted APIs.

Gateways such as LiteLLM, Portkey or Kong AI Gateway route requests between cloud model APIs and add keys, logging and sometimes guardrails, but they do not run models. OneAI runs models and routes to providers from the same process, so one policy, one cache and one audit trail cover both paths.

Fast on the second request, too.

Concurrent requests share one decode loop. A finished request keeps its KV cache, so the next turn only processes new tokens. An identical deterministic request replays from the response cache without loading the model.

Time to first token on an RTX 4080 SUPER · load modes: single, swap, multi · response cache: exact or semantic

Search and triage from the same GGUF files.

BERT-family encoders serve embeddings, classification and reranking. Vectors come back L2-normalized, ready for any retrieval stack.

Transcribe, translate and speak.

Whisper transcribes and translates. Piper and Kokoro read text aloud. Audio workers run as private processes with no open port.

Agents can call tools on local models.

Send OpenAI tools to a local model. OneAI parses the model's tool calls out of the stream and returns standard tool_calls deltas. Tool results go back on the next turn, and a routing rule can send tool requests wherever you choose.

# example stream · local model
data: {"choices":[{"delta":{"tool_calls":[{"index":0,"type":"function",
  "function":{"name":"lookup_order","arguments":"{\"order_id\":\"4471\"}"}}]}}]}

data: {"choices":[{"delta":{},"finish_reason":"tool_calls"}]}

Know where the time went.

Each request is one trace, with spans for routing, model load, queueing, prefill, generation and every provider attempt. Export stays off until you turn it on.

Scrape it like anything else.

Requests, tokens, prompt-cache reuse, provider health and spend.

# example scrape
oneai_active_requests 3
oneai_loaded_models 2
oneai_prompt_tokens_total 1284112
oneai_prompt_tokens_cached_total 902331
oneai_completion_tokens_total 412907
oneai_errors_total 21
oneai_gateway_external_providers 1
oneai_proxy_provider_health{provider="deepseek"} 2

Watch the hardware work.

CPU, memory, GPU, disk and network for the server machine, with OneAI's own share shown separately.

Performance · actual interface, dark theme

Usage you can explain.

Tokens, requests, cache hits, errors and spend across local and provider traffic, by model and by day. Download any view as CSV.

Activity · demonstration data, not a benchmark