AI Infra Bench
AI Infra Bench (AIB) is a command-line tool for inference services. Use an OpenAI-compatible endpoint to measure latency and throughput, evaluate model answers, and find a load limit that meets your service-level objectives (SLOs).
No serving-framework SDK is required. Start your inference service, then run aib from a client machine.
Choose a task
| I want to | Start here |
|---|---|
| Install AIB and run a first test | Quickstart |
| Check that a model responds | Send requests |
| Compare performance across concurrency levels | Benchmarking |
| Find capacity within a latency target | SLO search |
| Replay conversation workloads | Session replay |
| Measure answer accuracy | Evaluation |
| Inspect SGLang server metrics | Local monitoring |
Endpoint requirements
Text requests use /v1/chat/completions; random-token benchmarks use /v1/completions. Benchmarks require SSE streaming. Token usage, cache metrics, and speculative-decoding metrics depend on what the server reports.
Random token IDs, ignore_eos, extended statistics fields, and /flush_cache are not supported by every OpenAI-compatible service. Begin with a plain text request, then enable benchmark features. See Troubleshooting.