Skip to main content

AI Infra Bench

AI Infra Bench (AIB) is a command-line tool for inference services. Use an OpenAI-compatible endpoint to measure latency and throughput, evaluate model answers, and find a load limit that meets your service-level objectives (SLOs).

No serving-framework SDK is required. Start your inference service, then run aib from a client machine.

Choose a task​

I want toStart here
Install AIB and run a first testQuickstart
Check that a model respondsSend requests
Compare performance across concurrency levelsBenchmarking
Find capacity within a latency targetSLO search
Replay conversation workloadsSession replay
Measure answer accuracyEvaluation
Inspect SGLang server metricsLocal monitoring

Endpoint requirements​

Text requests use /v1/chat/completions; random-token benchmarks use /v1/completions. Benchmarks require SSE streaming. Token usage, cache metrics, and speculative-decoding metrics depend on what the server reports.

Compatibility

Random token IDs, ignore_eos, extended statistics fields, and /flush_cache are not supported by every OpenAI-compatible service. Begin with a plain text request, then enable benchmark features. See Troubleshooting.