CLI reference
Run aib --help to list commands and aib <command> --help for the complete options in your installed version.
Commands
| Command | Purpose |
|---|---|
req | Send one request |
bench | Benchmark or replay independent requests |
slo | YAML-driven SLO search |
session-bench | Replay sessions; alias session-reply-bench |
eval-dataset | Evaluate dataset accuracy |
eval-logits | Compare endpoint Top-K logprobs |
eval-hidden-states | Compare hidden-state extension outputs |
plot-metrics | Export aggregate metrics to interactive HTML |
monitor | Manage a local monitoring stack |
reply | Replay token-ID input/output data |
check-device | Inspect NVIDIA GPU hardware properties without torch |
check-weight | Inspect safetensors model weights |
profile-sgl | Start or stop SGLang profiling |
token | Encode/decode using a Hugging Face tokenizer; alias tokenize |
Some advanced commands need server extensions or additional dependencies. Check their individual help before use.
GPU hardware inspection
aib check-device # All CUDA-visible GPUs (default)
aib check-device --device 0 # One CUDA device ordinal
aib check-device --device 0 --device 1 # Multiple devices
aib check-device --json # JSON output
Uses only the Python standard library and NVIDIA driver; no extra Python dependencies, torch, or CUDA Toolkit.
Reports compute capability (sm_XX), SM count, shared memory limits per block/SM, L1 support, L2 capacity, memory capacity/bus width, clock rates/periods, and estimated memory bandwidth.
CUDA provides no portable L1 capacity query, so that field is N/A (null in JSON), as are unsupported attributes.
Clocks are driver-reported device attributes, not live samples; periods are inverse frequencies, not instruction latencies.
Bandwidth uses the DDR estimate 2 × memory clock × bus width / 8, not a measurement; that assumption may not match every memory technology.
Device ordinals respect CUDA_VISIBLE_DEVICES and may differ from nvidia-smi. Missing drivers, no visible GPUs, and invalid indices produce a nonzero exit status.
CUDA Core (FP32 lane) and Tensor Core counts are estimated from known per-SM configurations and the visible SM count.
Theoretical throughput includes CUDA FP32 and Tensor TF32, FP16, BF16, FP8 (TFLOPS), and INT8 (TOPS), with separate dense and 2:4 structured-sparse peaks.
Floating-point Tensor modes use FP32 accumulation; INT8 uses INT32 accumulation. FMA counts as two operations.
The formula is SM count × operations per SM per cycle × clock(kHz) / 10^9; this is not a benchmark.
Tensor profiles cover V100, T4, A100/A800/A30, A10/A40, L4/L40/L40S, and H100/H200/H800. Unmatched products and unsupported precisions display N/A.
L40 and L40S have the same SM version but different Tensor throughput and use separate profiles. Unknown SM versions do not inherit core counts from nearby architectures.
Common connection options
These options apply to req, bench, eval-logits, session-bench, and HTTP dataset evaluation. eval-logits requires one or more explicit addresses using --base-urls, also accepting the alias --base-url.
| Option | Default | Description |
|---|---|---|
--base-url | 127.0.0.1:30000 | Service root URL; trailing /v1 accepted |
--api-key | OPENAI_API_KEY or JustKeepMe | Explicit CLI value takes precedence over the environment |
--model | Unset | Model name; required if your server requires a model field |
--override-payload | Unset | JSON object overriding top-level payload fields |
--seed | 42 | Random request generation seed; not used by eval-logits |
req, bench, and eval-logits all accept --prompt (alias --user-prompt) and --output-len (alias --max-tokens). Text prompts use chat completions. Bench repeats the prompt for 100 requests by default, mutually exclusive with dataset and payload files. --input-len does not resize text prompts. Repeated prompts may hit the server's prefix cache.
Payload precedence: input body/defaults < explicit options < --override-payload. Overrides must be JSON objects; merging is shallow, replacing entire nested objects. Streaming follows command requirements: bench uses streaming statistics, req switches with --disable-stream, and eval-logits requires non-streaming responses and rejects stream:true.
SLO configures connections in the YAML endpoint section instead of these CLI options.
Common bench options
| Option | Default | Description |
|---|---|---|
--dataset | random | Mutually exclusive with --payload-regex-path |
--num-requests | 100 for random/ShareGPT | Other workloads use the available data count |
--max-concurrency | 32 | Accepts multiple positive integers |
--request-rate | inf | Request arrival rate |
--input-len / --output-len | 8192 / 1024 | Target lengths for random/ShareGPT |
--num-warmup-requests | 10 | Maximum warmup requests |
--repeat | 1 | Repeats per concurrency setting |
--metric-path | aib-metric.jsonl | Aggregate metrics file |
--disable-warmup | Off | Skip ordinary warmup |
--disable-flush-cache | Off | Skip cache flushing and bench warmup |
Token utility
python -m pip install 'ai-infra-bench[data]'
aib token --tokenizer /path/to/tokenizer "Hello world"
aib token --tokenizer /path/to/tokenizer --decode '[1, 2, 3]'
Token IDs depend on the model and tokenizer. Use the same tokenizer as your server.