SLO search
SLO search asks: “How much load can this service handle while meeting success-rate and latency targets?” AIB uses binary search over an integer range defined in YAML and records every probe.
Create a configuration
Save this as slo.yaml, replacing the endpoint and model:
endpoint:
base_url: http://127.0.0.1:30000
api_key: EMPTY
model: your-model
request:
payload:
messages:
- role: user
content: Explain prefix caching in one paragraph.
max_tokens: 128
benchmark:
num_requests: 100
warmup_requests: 5
max_concurrency: 32
request_rate: inf
seed: 42
search:
parameter: max_concurrency
min: 1
max: 64
conditions:
- metric: success_rate
operator: ">="
value: 0.99
- metric: p99_latency_ms
operator: "<"
value: 3000
aib slo slo.yaml -o slo-result.yaml
Every condition must pass. success_rate is a fraction from 0 to 1; latency is in milliseconds.
Search request rate
Change search.parameter to request_rate and set integer min and max values in requests per second. benchmark.max_concurrency then stays fixed. When searching concurrency, the fixed rate comes from benchmark.request_rate.
Comparison operators are <, <=, >, >=, and ==; quote them in YAML. See Metrics and results for supported metrics.
Random-token mode
Replace request.payload with:
request:
input_len: 1024
output_len: 256
This uses completions and requires token-ID and ignore_eos support. Set endpoint.path for a custom API path.
Read the report
The report includes status, search_parameter, best_value, and runs. Each run records the candidate, metrics, and condition evaluations. If no probe passes, best_value is null and status is unsatisfied. This does not cause a nonzero command exit code; automation must inspect the report status.
Binary search assumes that higher load makes the objectives harder to meet; it does not exhaustively test every integer. Cache state, warmup, and small samples can affect that assumption. Treat the result as candidate capacity for this workload and repeat benchmarks near the boundary.