Skip to main content

CLI reference

Run aib --help to list commands and aib <command> --help for the complete options in your installed version.

Commands​

CommandPurpose
reqSend one request
benchBenchmark or replay independent requests
sloYAML-driven SLO search
session-benchReplay sessions; alias session-reply-bench
eval-datasetEvaluate dataset accuracy
eval-logitsCompare endpoint Top-K logprobs
eval-hidden-statesCompare hidden-state extension outputs
plot-metricsExport aggregate metrics to interactive HTML
monitorManage a local monitoring stack
replyReplay token-ID input/output data
check-deviceInspect NVIDIA GPU hardware properties without torch
check-weightInspect safetensors model weights
profile-sglStart or stop SGLang profiling
tokenEncode/decode using a Hugging Face tokenizer; alias tokenize

Some advanced commands need server extensions or additional dependencies. Check their individual help before use.

GPU hardware inspection​

aib check-device                         # All CUDA-visible GPUs (default)
aib check-device --device 0 # One CUDA device ordinal
aib check-device --device 0 --device 1 # Multiple devices
aib check-device --json # JSON output

Uses only the Python standard library and NVIDIA driver; no extra Python dependencies, torch, or CUDA Toolkit. Reports compute capability (sm_XX), SM count, shared memory limits per block/SM, L1 support, L2 capacity, memory capacity/bus width, clock rates/periods, and estimated memory bandwidth. CUDA provides no portable L1 capacity query, so that field is N/A (null in JSON), as are unsupported attributes. Clocks are driver-reported device attributes, not live samples; periods are inverse frequencies, not instruction latencies. Bandwidth uses the DDR estimate 2 × memory clock × bus width / 8, not a measurement; that assumption may not match every memory technology. Device ordinals respect CUDA_VISIBLE_DEVICES and may differ from nvidia-smi. Missing drivers, no visible GPUs, and invalid indices produce a nonzero exit status.

CUDA Core (FP32 lane) and Tensor Core counts are estimated from known per-SM configurations and the visible SM count. Theoretical throughput includes CUDA FP32 and Tensor TF32, FP16, BF16, FP8 (TFLOPS), and INT8 (TOPS), with separate dense and 2:4 structured-sparse peaks. Floating-point Tensor modes use FP32 accumulation; INT8 uses INT32 accumulation. FMA counts as two operations. The formula is SM count × operations per SM per cycle × clock(kHz) / 10^9; this is not a benchmark. Tensor profiles cover V100, T4, A100/A800/A30, A10/A40, L4/L40/L40S, and H100/H200/H800. Unmatched products and unsupported precisions display N/A. L40 and L40S have the same SM version but different Tensor throughput and use separate profiles. Unknown SM versions do not inherit core counts from nearby architectures.

Common connection options​

These options apply to req, bench, eval-logits, session-bench, and HTTP dataset evaluation. eval-logits requires one or more explicit addresses using --base-urls, also accepting the alias --base-url.

OptionDefaultDescription
--base-url127.0.0.1:30000Service root URL; trailing /v1 accepted
--api-keyOPENAI_API_KEY or JustKeepMeExplicit CLI value takes precedence over the environment
--modelUnsetModel name; required if your server requires a model field
--override-payloadUnsetJSON object overriding top-level payload fields
--seed42Random request generation seed; not used by eval-logits

req, bench, and eval-logits all accept --prompt (alias --user-prompt) and --output-len (alias --max-tokens). Text prompts use chat completions. Bench repeats the prompt for 100 requests by default, mutually exclusive with dataset and payload files. --input-len does not resize text prompts. Repeated prompts may hit the server's prefix cache.

Payload precedence: input body/defaults < explicit options < --override-payload. Overrides must be JSON objects; merging is shallow, replacing entire nested objects. Streaming follows command requirements: bench uses streaming statistics, req switches with --disable-stream, and eval-logits requires non-streaming responses and rejects stream:true.

SLO configures connections in the YAML endpoint section instead of these CLI options.

Common bench options​

OptionDefaultDescription
--datasetrandomMutually exclusive with --payload-regex-path
--num-requests100 for random/ShareGPTOther workloads use the available data count
--max-concurrency32Accepts multiple positive integers
--request-rateinfRequest arrival rate
--input-len / --output-len8192 / 1024Target lengths for random/ShareGPT
--num-warmup-requests10Maximum warmup requests
--repeat1Repeats per concurrency setting
--metric-pathaib-metric.jsonlAggregate metrics file
--disable-warmupOffSkip ordinary warmup
--disable-flush-cacheOffSkip cache flushing and bench warmup

Token utility​

python -m pip install 'ai-infra-bench[data]'
aib token --tokenizer /path/to/tokenizer "Hello world"
aib token --tokenizer /path/to/tokenizer --decode '[1, 2, 3]'

Token IDs depend on the model and tokenizer. Use the same tokenizer as your server.