Diagnostic loop for vLLM on one GPU. Deterministic engineering: same server, same traffic, same verdict. Close the gap in a few measured iterations instead of guessing for days.
In one sitting
Install the binary
Run profile diagnose under live traffic
Apply the Fix, press Enter, read the delta
Repeat until the loop names a wall or goes quiet
Profile is not a dashboard and not an autotuner. It computes the hardware ceiling for your model and GPU, measures live throughput against it, names the one cause holding the server back, and gives you the vLLM flags to change. You apply the fix; Profile never restarts your server. No calibration run.
Start here, then Rules for thresholds. The rest is reference.
Prerequisites
One GPU: NVIDIA (NVML) or AMD (amdgpu). Profile probes NVIDIA first and falls back to AMD. --tensor-parallel-size greater than 1 is refused today.
vLLM with /metrics reachable (default http://localhost:8000/metrics).
Live traffic during the window. An idle server has no waste to find; Profile says so. Drive load with vllm bench serve if you need traffic.
Install and diagnose
# Install
curl --proto '=https' --tlsv1.2 -LsSf \
https://github.com/jungledesh/profile/releases/latest/download/profile-installer.sh | sh
# Diagnose (default window is 30s; raise when traffic cycles inside the window)
profile diagnose --url http://localhost:8000/metrics --duration 30s
Or build from source: cargo install --git https://github.com/jungledesh/profile. No curl-pipe: download the binary from the releases page.
Duration and traffic shape
Match --duration to the cycle. Default for steady load. Raise it when traffic repeats inside the window. Do not raise it when load changes between iterations.
Shape
What it looks like
--duration
Steady
Load holds still
Default 30s is enough
Fast bounce
Wobble shorter than a slice (2s at 30s, 10s above)
Already averaged. Leave it
Cycle near the window
Agents start, time out, and restart together
Raise so several cycles fit, up to 30m. A ~10 min cycle needs about 30m
Step between iterations
Flat during a run, different on the next
Not a duration problem
Optimization Workflow
When sampling ends, Profile prints a summary block. Look at ISSUES. The Fix: tells you what to change in your vLLM startup command.
Read
Profile prints a performance snapshot followed by an ISSUES block. Here is an example. Do exactly what the Fix: section recommends.
+----------------------------------------------------------------------------------------------------------------------+
|PROFILE v2.2.1 [muse-glimmer-30b] [NVIDIA GeForce RTX 5090] (5m from 2026-08-14 10:30:26 UTC) |
| |
|GPU => decode_eff ~3.5% | power 531W | 4.05 J/tok | $2.10/1M output tok (est) | vRAM 29/32GB |
| mem_util 37% |
| |
|vLLM => |
|REQUESTS run 9 (27.4%) | wait 15 | max 32 |
|LATENCY ttft 32.8s (p95 66.9s) | tpot 58ms (p95 89ms) |
|CACHE kv_cache 88.6% avg (99.9% peak) | pfix_cache 22.7% |
|THROUGHPUT 131 tok/s |
|TRAFFIC qps 0.5 | req_total 642 | gen_total 175857 | preempt/s 0.16 | preempt_total 59 |
| |
|ISSUES: |
| |
|[!] KV Cache Pressure |
| Seen in 100% of windows |
| Cause: |
| KV cache 89% avg in fired windows, 100% peak (threshold: 88%). |
| Scheduler evicting; 15 requests queued on KV admission. |
| |
| Fix: |
| • Raise --gpu-memory-utilization (check vRAM header for avail mem) to expand KV pool. |
| • Switch --kv-cache-dtype fp8 to halve KV memory footprint (affects output quality). |
| • Lower --max-num-seqs to reduce KV demand. |
| Cuts throughput. Revert after pressure clears. |
| • Reduce client concurrency toward sustained running (9 in-flight). |
| Cuts queue wait, not throughput. Demand exceeds admitted capacity. |
| • Lower --max-model-len 32768 → 21933. Observed p99 21.9k tokens per request. |
| ~1% of observed requests ran longer; those are rejected with a 400, not truncated. |
| |
| Expected: TTFT and TPOT recover once evictions stop. |
| Confidence: High |
+----------------------------------------------------------------------------------------------------------------------+
Every value is measured or marked. A dash (-) means the metric was not readable. (est) means the number came from the physics model. A tilde (~) marks a value derived from an estimated ceiling. Gaps are never filled with guesses.
Note: [!] KV Cache Pressure corresponds directly to rule R2 in the Flag Recommendations Map below.
Restart & Measure
Apply primary and secondary fixes together in one restart. One flag per restart wastes time. Restart vLLM. Profile resumes on vLLM re-start and measures the new baseline. Repeat this process until the bottleneck clears or you reach hardware saturation.
Apply the fix above. Profile re-measures after your change.
Press Enter when done.
Connection restored. Resuming in 5s...
New --max-num-seqs [current: 32]: 12
Measuring delta...
Config changed. Baseline reset.
Throughput 131 → 421 tok/s
TTFT 32857 → 224ms (p95 66946 → 500ms)
TPOT 58.0ms → 23.0ms (p95 58.0 → 40.0ms)
ECONOMICS:
J/tok 4.05 → 1.08
Cost/1M output tok $2.10 → $0.65 (est)
Notice the throughput dropped? Profile reports regressions honestly. Fixing one bottleneck often exposes the next. Keep iterating; results improve over multiple restarts.
Loop behavior
Same-primary reveal. When the same primary fires again, Profile lists the rules it suppressed under that block. No flat-delta gate: re-fire alone is enough, whether or not you applied a fix in between.
Prefill unread re-fire exit. When Prefill is primary twice in a row and both Fix blocks include the unread --max-num-batched-tokens guide (common when vLLM never emits the gauge), the second table still prints, then the loop exits: no new server lever to apply. First unread show stays open (Confirm + guide). Bound Severe/compute-wall terminals are unchanged.
Empty-fix reveal. When the primary has no lever left (fix list empty or exhausted), Profile shows the suppressed alternatives under that same block immediately, without waiting for a second fire.
Dead-end. When no config lever remains, Profile names the wall (replica, or the physics boundary), points at scale-out, and surfaces what it was holding.
Oscillation escape. When R2 (KV pressure) and R5 (concurrency saturation) alternate on --max-num-seqs, Profile names the bracket tried and suggests the midpoint. It offers this at most three times, then names the wall.
Config-skip. Where Profile can read your running config, it skips levers you already set: prefix caching on, fp8 KV already active, chunked prefill on, --max-num-batched-tokens already at or above the suggestion, seats already at the target. Exception: --kv-offloading-size is re-derived on each R2 fire; if the new size differs from what is set, the flag is offered again with the new number (host-RAM subline included). If derived equals set, it stays quiet.
Quiet end
When no rule fires, Profile names the cap and stops inventing flags. RTX 5090 Muse run, after the KV fix:
Profile detects these bottlenecks and recommends the following vLLM flag changes:
Diagnosis
When it fires
vLLM flags to change
R1 Under-batching
Known GPU: config-relative efficiency <60% with occupancy <75%, no backlog. Occupancy denom is min(max_num_seqs, ridge, observed kv_cache_max_concurrency). Unknown GPU: occupancy <25%. Soft field: ranks above Prefill/Prefix on first fire.
Batch more requests or raise client concurrency
R2 KV cache pressure
KV ≥88% (avg or peak) AND (eviction: preempt/s >0.02 or swapped ≥2, OR waiting >2)
KV near full (≥88% avg or peak), queue ratio ≥30%, scheduler not at seat cap, free KV below demand. May fire on the same windows as R2; when both are significant, R2 wins at report construction.
Expand KV pool: raise --gpu-memory-utilization, fp8 KV, or reduce --max-model-len
R3 Low prefix reuse
Active traffic (running >0.75), mean prompt ≥20 tok, QPS × mean prompt ≥1000. Caching off: fires on volume alone. Caching on: hit rate <35%.
--enable-prefix-caching; restructure prompts if already on
R4 OOM risk
Weights overflow VRAM, or free VRAM cannot hold one worst-case request. Not fired when dtype is fallback (bf16 assumed).
--tensor-parallel-size at minimum needed, raise --gpu-memory-utilization, or use a smaller model
R5 Concurrency saturation
Collector seat_wall_cooccurred (a scrape with running in the churn band at --max-num-seqs and waiting ≥2; slack = clamp(completions/s × 250ms, 1, max(0.01 × max, 1))), plus window waiting ≥2 and queue ratio ≥30%. No peak/3% fallback: flag unread or false → silent.
Below 80% KV: raise --max-num-seqs to bounded target. At/above 80% KV: name the wall or add a replica.
R6 Prefill-bound
Effective prompt/gen ratio ≥5 (prefix hits removed from prompt tok/s), decode efficiency <40%. Muted when TPOT is measured and under 4× its floor. Bound path silences R1; soft field does not.
Prefix caching first when confirmed off. Then chunked prefill; --max-num-batched-tokens (no blind Set when unread); at compute wall, disaggregate or add a replica
R7 Config headroom
--max-num-seqs below 90% of recommended target, occupancy ≥50%, ≤1 waiting
Raise --max-num-seqs; output names the binding wall
Note: Profile may list several fixes in one Fix: block. Apply them together when relevant. See Rules for thresholds and edge cases.
Profile CLI Configuration
These are flags for the profile CLI itself, not vLLM. Each also has an env var (PROFILE_URL, PROFILE_DURATION, PROFILE_MAX_NUM_SEQS, PROFILE_TENSOR_PARALLEL_SIZE, PROFILE_COST_PER_HOUR, PROFILE_VERBOSE).
Flag
Default
Description
-u, --url
http://localhost:8000/metrics
vLLM metrics endpoint
--duration
30s
Collection window per iteration. Minimum 30s, maximum 30m. Units s or m only, not ms/mins. Match to traffic shape (above).
-m, --max-num-seqs
Prompted if unread
Skip the prompt. Preflight reads /metrics when the gauge is present; otherwise you are asked.
--tensor-parallel-size
Unset
Must be 1. Values above 1 are refused. Pass 1 to skip the GPU-assignment prompt. Profile still reports when a model needs TP to fit at all.
--cost-per-hour
Catalog estimate
GPU cost in USD/hr. Must be a positive number.
-v, --verbose
Off
Rules that did not fire, physics limits, and extra GPU, latency, cache, and config detail
profile help, profile completions <SHELL>, and profile man exist. They are not diagnose flags.
H100 80GB HBM3 · Qwen3.8-27B:257 → 490 tok/s (1.9x). Cost $3.23 → $1.69 / 1M tok (48% lower). TTFT 1.9s → 539ms. Path 257, 278, 490. Raising agents without a KV fix sent TTFT to 172s; Profile labelled it worse, then prefix + fp8 KV + seats 22 recovered it. Ended capped by traffic. Every iteration · Video
The flood is in the record on purpose. A server already near its ceiling has nothing to recover, and Profile says so instead of inventing a flag.
Why Profile?
For everyone running vLLM on one GPU. The value is time: a few measured iterations instead of days of guessing. Profile plugs into the serving stack; it replaces nothing. Not yet: multi-GPU sharding, and engines beyond vLLM.
Profile
Dashboards
Kernel profilers
Autotuners
Simulators
Hardware ceiling from physics
yes
no
no
no
predicted
Live server, production traffic
yes
yes
yes
no
no
Names one root cause
yes
no
no
no
no
Prescribes the change
yes
no
no
config only
config only
Measures the delta after the fix
yes
no
no
partial
no
Cost per million tokens
yes
no
no
no
no
No auto-restart, no synthetic load
yes
yes
yes
no
n/a
Rules
Eight rules (R1–R7, R2b sibling). Each fires when a specific bottleneck is confirmed under load.
Rules evaluate on structurally valid windows (window_is_evaluable). A recommendation surfaces only when the signal is persistent: at least 3 windows and at least 25% of all evaluable windows in the run.
Display metrics (like requests_running and requests_waiting) are synced to the final session snapshot rather than locally aggregated window approximations, to prevent visual contradictions between the summary header and the recommendation blocks.
R1: Under-batching
Condition
Threshold
Known GPU: config-relative efficiency
<60% of config capacity (decode ceiling × min(max_num_seqs, ridge))
Known GPU: occupancy
<75% of effective max
No backlog
waiting < 2
Unknown GPU fallback: occupancy
<25% of effective max (same binder as known-GPU: min of config, ridge, observed KV)
R1 still evaluates when R6 (Prefill-bound) fires on the same window. On a pressed box (bound path), R6 suppresses R1 via the mutual exclusivity table so under-batching is held for reveal, not silently deferred. Under soft field (waiting < 2, KV mean < 80%, running well below the decode ridge Traffic floor), that ME row is skipped: R1 owns first fire; Prefill/Prefix land in suppressed_recs for same-primary remeasure reveal only. Fix is client-side (batch more requests / raise concurrency) and names the binding wall: config cap, compute ridge, or memory limit.
Fix: Batch more requests or increase client concurrency.
R2: KV cache pressure
R2 and R2b are sibling rules in the same DAG layer. Both surface KV-related pressure. Both detectors may count the same window; when both are significant at report time, construction keeps R2 and skips R2b (else if), not the mutual exclusivity table. The ME table silences R2/R2b under R4 weights-alone overflow, and R1 under R6 on the bound path only (soft field skips that row; see Ranking).
Main path (R2)
Condition
Threshold
KV near full (avg or peak)
≥88%
Harm signal (at least one)
preempt/s > 0.02, or swapped ≥ 2, or waiting > 2
Both conditions must hold. KV at 88% with no evictions and no queue is healthy and hot; R2 stays quiet.
The fix block is split into Cuts throughput and Safe to apply. The seat direction line (--max-num-seqs) gives direction only, not a specific ceiling number. --max-model-len shrink prefers observed p99 prompt + p99 gen; falls back to averages when p99s are missing.
--kv-offloading-size: on every R2 fire, Profile re-derives the size (supply-capped by host / container RAM). If the derived value differs from what is already set, the Safe block offers the new number (e.g. set 4, derived 12 → Set --kv-offloading-size 12 (est)) with the Host RAM subline. If derived equals set, the lever stays quiet.
Confidence:windows_fired / total_evaluable. Ranges from Low to High as more windows confirm the signal.
Admission backlog path (R2b)
Condition
Threshold
KV near full (avg or peak)
≥88% (same bar as R2)
Queue ratio
waiting / (running + waiting) ≥ 30%
Free KV tokens
< demand from queued requests (waiting × prompt_tokens_mean)
Concurrency cap
running < max_num_seqs (scheduler not at cap)
Fires when KV is near full and the scheduler is holding requests in queue to protect KV memory. Both R2 and R2b may count the same window; when both are significant, report construction keeps R2 and skips R2b (else if). The fix is to expand the KV pool, not reduce concurrency.
Fix: Raise --gpu-memory-utilization if VRAM headroom exists; switch to fp8 KV cache; or reduce --max-model-len.
KV near-full uses avg or peak ≥88%. Peak matters because a window can average 70% and still spike to 92%: the spike is what causes preemptions; the average alone can hide it. Peak is also what the header shows when it exceeds avg by ≥10pp or reaches 95%.
R3: Low prefix reuse
Condition
Threshold
Active traffic: running
>0.75
Mean prompt tokens
≥20
Prompt token throughput (QPS × mean prompt)
≥1000 tok/s
Prefix caching off
Fires on volume alone (hit rate not required)
Prefix caching on: hit rate
<35%
Confidence: 0.95 when prefix caching is off (direct fix available); 0.90 when caching is on and hit rate is low.
Fix: If prefix caching is disabled: enable --enable-prefix-caching. If already enabled: move shared instructions to the start, standardize templates, avoid unique tokens at the beginning.
R4: OOM risk
Fires regardless of traffic. Configuration fact, not runtime observation. Does not fire when dtype is bf16 fallback (insufficient information to prescribe confidently).
Two fire paths: (1) kv_headroom_gb < 0, weights overflow. (2) Weights fit but free VRAM cannot hold one worst-case request's KV + state.
Fix:--tensor-parallel-size at the computed minimum; raise --gpu-memory-utilization; use a smaller model; or the model does not fit on this hardware.
The 3.0 GB buffer reserves activation memory. kv_headroom_gb negative means the model cannot run at all on this TP configuration, not just that KV space is tight.
R5: Concurrency saturation
Condition
Threshold
Seat wall co-occurrence
Collector seat_wall_cooccurred: any 250ms scrape with running in the churn band at max_num_seqs and waiting ≥ 2. Slack = clamp(completions/s × 250ms, 1, max(0.01 × max, 1)). Samples above max + 0.5 are evaluated-false (chunked cross-step batching), not a hit. Zero evaluable samples or flag unread/false → R5 silent. No peak/3% fallback.
Queue ratio (window)
waiting / (running + waiting) ≥ 30%
Waiting floor (window)
≥ 2
R5 fires without a KV <80% gate. The 80% check is in the fix branch only: below it, Profile raises --max-num-seqs to a bounded target; at or above it, Profile names the wall and recommends a replica instead.
Fix: Below 80% KV: raise --max-num-seqs to the bounded target (80% margin on the binding wall, then capped by a live-traffic floor when one exists). Above 80% KV: name the wall (replica; or lower --max-model-len to shrink KV demand).
Muted when TPOT is measured and under 4× its floor. Still fires when TPOT or floor is missing (confidence capped at 0.5).
R6 fires at DAG layer 5. Severity from effective prompt/gen ratio: Mild ≥5, Moderate ≥10, Severe ≥20 (confidence tiers 0.65 / 0.75 / 0.85; capped at 0.5 when TPOT unverified). On the bound path, the suppression table silences R1 (Under-batching) so a pressed, prefill-bound server does not get “send more traffic” as primary; R1 still evaluates into suppressed_recs for reveal. Under soft field, that ME row is skipped so R1 owns first fire and Prefill is held for remeasure reveal (light-load Prefill is not the setup wall).
Fix: Prefix caching first when confirmed off (Some(false)). Enable chunked prefill only when confirmed off; Confirm when unread (never Enable on unknown). When configured --max-num-batched-tokens is readable and below the launch default (2048), Set to 2048 plus optional page floor; never Set down when already above default; when unread, guide with floor/directions (no blind Set). At the compute wall (configured within 20% of a derived ridge/workload recommendation) or Severe FLOPs wall (ratio ≥ 20): disaggregate or add a replica (terminal when no local knob remains), and show suppressed alternatives under the same block when applicable.
R7: Config headroom
Condition
Threshold
--max-num-seqs vs recommended target
<90% of the target
Occupancy
≥50% of configured max_num_seqs (running / max_num_seqs)
Waiting
≤1
Optimistic rule (DAG layer 6). Only fires when no active bottleneck is present. The recommended target is shared with R5: 80% margin on the binding wall (ridge or observed KV capacity). Output names what binds the target.
Fix: Raise --max-num-seqs to the computed target.
Confidence
Per-rule, not a unified formula.
Rule
Confidence
Condition
R1
0.80 (known GPU) / 0.50 (unknown)
Fixed per GPU-knowledge path
R2 / R2b
windows_fired / total_evaluable
Density: rises as more windows confirm the signal
R3
0.95 (caching off) / 0.90 (caching on, low hit rate)
Fixed per caching path
R4
0.95
dtype or quantization from env var (DTYPE/VLLM_DTYPE, QUANTIZATION/VLLM_QUANTIZATION)
R4
0.90
vLLM-reported dtype or quantization, or catalog default
R4
Not fired
weight dtype is bf16 fallback
R5
0.50
Empirical KV bound used
R5
0.90
TTFT and KV cache both present
R5
0.60
One or both absent
R6
0.85 / 0.75 / 0.65 by severity; capped at 0.5 when TPOT unverified
Shared by windowed rules (R1–R3, R5–R7, R2b). A rule that fired once is noise. R4 is exempt: it is a config-fact path from baseline headroom, not a per-window fire count, so it skips this gate.
Rules are ranked by score within the winning DAG layer. The highest-scoring rule in the winning layer is the primary recommendation. All others are held. They surface when the same primary re-fires, or immediately when the primary has no fix left to offer.
DAG layers: R4, R2, R2b at layer 2 (highest priority). R5 at 3. R1 at 4. R3 and R6 at 5. R7 at 6 (lowest). The winning layer is the lowest-numbered layer that has a significant rule. Rules in higher-numbered layers are suppressed when a lower layer wins.
Mutual exclusivity: R4 (weights overflow alone) silences R2 and R2b. R6 (Prefill-bound) silences R1 (Under-batching) on the bound path. Soft field (waiting < 2, KV mean < 80%, running < max(0.25 × ridge, 8), same floor as the Traffic limiter): skip that R6→R1 row so Under-batching owns first fire; Prefill and Prefix fall to suppressed_recs for same-primary remeasure reveal. Missing soft-field inputs keep bind-path ME. These suppressions run before the layer filter. R2 vs R2b is handled separately in eval (else if), not this table.
When no rule fires
If efficiency is still low, a diagnose-only fallback names the shape of the underuse (soft under-fed inject can also reuse R1's fix when R1 gates miss but the field is soft and Prefill would own the page). If nothing fires at all, Profile names the boundary capping a healthy server (PrimaryLimiter). Verdicts use run-level aggregates, never one snapshot.
Capacity (memory):Capped by memory: KV cache at N% avg (R2 fires at 88%). Concurrency cannot grow further on this pool.
Traffic:Capped by traffic: N requests running, no queue. Raise client concurrency in steps; compute ridge ~R, but KV or admission can bind first. Ridge is the decode compute knee, not a seat target.
Physics (hardware):Capped by hardware: TPOT within 1.2× its floor, or efficiency headroom below 10%. Scale out.
Prefill interference:Capped by prefill: prompt work at Nx of decode (effective). Requires measured effective prompt/decode ratio ≥ 0.5; chunked prefill is a precondition, not evidence.
Framework overhead:Capped by vLLM overhead: TPOT above floor with no other healthier name. Requires measured TPOT.
Waiting unread:Waiting unread; cannot name a healthy cap. Not a Capped by line.
Mean waiting ≥ 2: no healthy Capped by over a flood; rules own the story, or silence.
Speculation suspected: decline a hardware Physics cap. Scoreboard: Note: Throughput above the decode ceiling (speculative decoding likely). Efficiency % does not apply. Limiter / healthy exit: same body without Note:.
Unknown ceiling:Hardware ceiling unknown (...) instead of naming a boundary it cannot prove.
Data
Idle and dropped windows are excluded from active-window averages (throughput, latency, running). KV average uses all evaluable windows.
GPU and vLLM are scraped in parallel at ~250ms intervals inside each collection window. Window size: 2s when --duration is 30s or less; 10s otherwise. Each window produces one snapshot from ~9 or ~40 polls respectively.
The 1-second observation skew gate (|gpu_observed_at − vllm_observed_at| > 1s) applies only to energy and cost metrics. No rule skips based on skew.
Within each window, different metric types are treated differently:
Type
Treatment
Edge cases
Gauges
Last scrape in the window
kv_cache_peak_perc is max across all scrapes in the window, not last. KV cache can spike and recover within a window; last scrape alone misses it. Shown in output when peak > avg + 10pp or peak ≥ 95%. Same pattern for VRAM peak (≥90% of total) and GPU temp peak (≥80°C).
Histograms
Δsum/Δcount, first to last scrape
Falls back to last-scrape cumulative mean when Δcount ≤ 0 (no new completions in window). p99 has no fallback; stale p99 is worse than none.
Counters
Rates = Δ / window duration. Raw totals from last scrape.
None on counter reset (negative delta) or zero-duration window. Zero delta is valid: no activity, not missing data.
Evaluable window gate
Two-tier gate. Structural validity and active traffic are separate checks.
Tier 1: structural. Did the endpoint respond and does the window have a valid duration? All windowed rules (R1–R3, R5–R7, R2b) use this check. R4 is a static config-fact path and does not require an evaluable window.
window_duration_secs is present and positive
num_requests_running is Some (even if zero)
running = 0 is valid data: the server responded and reported no active requests. None means the endpoint did not respond.
Tier 2: active. Was the server doing real work? Aggregated means use this check. Throughput, efficiency %, GPU utilization, and latency averages are computed over active windows only.
window_is_evaluable(s) && !window_is_idle(s)
// idle: running < 1 AND generation_tokens_per_sec < 1 AND waiting < 1
// (None for any field counts as < 1)
Active is the inverse of idle for evaluable windows. Low numbers are included; blank numbers are not. Idle windows pass tier 1 but not tier 2 and are excluded from the averages.
Multi-window aggregation, which window set applies to which field, is in Math.
Source 1: GPU telemetry
NVIDIA via NVML, or AMD via libamdgpu_top when NVML is unavailable. Polled at 250ms intervals. Same gauge fields either way; FLOPs and bandwidth still come from the GPU catalog.
Field
Type
What it is
How collected
gpu_util_pct
Gauge
Fraction of time any kernel was executing. Not SM occupancy.
Mean across window polls
mem_util_pct
Gauge
Fraction of time the memory controller was busy
Mean across window polls
power_watts
Gauge
Power draw (W)
Mean across window polls
power_limit_watts
Gauge
Driver-set TDP limit (W)
Read when assembling each window's GPU row
vram_used_mb
Gauge
VRAM in use (MiB)
Last poll
vram_total_mb
Gauge
Total device VRAM (MiB)
Last poll
temperature_c
Gauge
GPU core temp (°C)
Last poll
sm_clock_mhz
Gauge
SM clock frequency (MHz)
Last poll
Two values are computed from the polls, not read directly: vram_peak_mb (max of vram_used_mb across all polls) and temperature_peak_c (max of temperature_c across all polls). Derived within the window, not collected.
Neither NVML nor the AMD path exposes theoretical peak FLOPs or memory bandwidth. Those come from the GPU catalog, looked up once at startup from gpu_name, labeled (est) in output. Host RAM (for --kv-offloading-size sizing) is read separately from the host / container memory path.
Source 2: vLLM Prometheus
Scraped from /metrics at ~250ms intervals. See vLLM docs for the full metric reference.
Static cache config: block_size, cache_dtype, prefix_caching, and sometimes chunked_prefill. Modern vLLM often omits chunked prefill and max-num-batched-tokens here; Profile then probes /info and /server_info best-effort. Missing fields stay unread, never guessed.
Model catalog. MoE: total for weight/OOM; active for roofline.
Roofline skipped
bytes_per_param (weights)
vLLM-reported quantization → QUANTIZATION/VLLM_QUANTIZATION env → vLLM-reported dtype → DTYPE/VLLM_DTYPE env → catalog default → bf16 (2 bytes). kv_cache_dtype is not in this chain; it sets KV element bytes only.
bf16 fallback, labeled in output
All catalog-derived numbers are labeled (est) in output. Upper-bound approximations, not measured values.
Fallbacks
Field
Primary
Fallback
If absent
max_num_seqs
vllm:max_num_seqs gauge
-m CLI flag
R1 and R5 silent
tensor_parallel_size
--tensor-parallel-size CLI flag
Treated as 1. Values above 1 are refused at launch; multi-GPU is not supported.
BF16 Tensor Core dense TFLOPs and memory bandwidth feed the roofline math. On-demand prices from src/context/gpu_prices.json (updated 2026-08-13). Use --cost-per-hour for exact rates. A dash means no catalog price.
GPU
Arch
BF16 TC Dense TFLOPs
Mem BW GB/s
On-demand $/hr
H100 PCIe
Hopper
756.0
2,000
$2.89
H100 NVL
Hopper
835.5
3,900
$3.19
H100 SXM
Hopper
989.0
3,350
$2.99
H200
Hopper
989.0
4,800
$4.39
A100 80GB
Ampere
312.0
2,039
$1.49
A100 40GB
Ampere
312.0
1,555
$1.50
A10G
Ampere
126.0
600
$1.00
L40S
Ada
181.03
864
$0.99
RTX 5090
Blackwell
209.5
1,792
$0.99
RTX 4090
Ada
165.0
1,008
$0.69
RTX 3090 Ti
Ampere
80.0
1,008
$0.22
RTX 3090
Ampere
71.16
936
$0.46
RTX A6000
Ampere
77.4
768
$0.49
RTX PRO 6000 Blackwell
Blackwell
250.0
1,792
$1.99
B200
Blackwell
2,250.0
7,700
$5.89
B300
Blackwell
2,500.0
8,000
$7.39
GB200
Blackwell
2,250.0
7,700
$8.50
GB10 (DGX Spark)
Blackwell
212.9
273
-
MI355X
CDNA4
2,500.0
8,000
-
MI350X
CDNA4
2,300.0
8,000
-
MI325X
CDNA3
1,307.4
6,000
$3.50
MI300A
CDNA3
980.6
5,300
$2.00
MI300X
CDNA3
1,307.4
5,300
$3.49
MI250X (per GCD)
CDNA2
191.5
1,638.4
$1.50
MI250 (per GCD)
CDNA2
181.0
1,638.4
$1.30
MI210
CDNA2
181.0
1,638.4
$1.00
RX 9070 XT
RDNA4
194.6
644
-
RX 9070
RDNA4
144.5
644
-
Radeon PRO W7900
RDNA3
123.0
864
-
Radeon PRO W7800
RDNA3
90.5
576
-
RX 7900 XTX
RDNA3
123.0
960
-
RX 7900 GRE
RDNA3
92.0
576
-
RX 7900 XT
RDNA3
103.0
800
-
RX 7800 XT
RDNA3
74.6
624
-
RX 7700 XT
RDNA3
70.3
432
-
RX 7600 XT
RDNA3
45.1
288
-
BF16 TC Dense TFLOPs are tensor-core throughput with FP32 accumulate where the catalog distinguishes it (L40S uses 181.03; datasheet 362.05 is FP16-accumulate). GB200 matches B200 physics (superchip name). B300 is Blackwell Ultra at 8 TB/s. H100 NVL is the 94 GB HBM3 SKU (3.9 TB/s). GB10 BW is LPDDR5X system bandwidth, not HBM. MI250 / MI250X values are per GCD (ROCm sees each GCD as a device). A plain "A100" without a size token (80GB or 40GB) will not match. All catalog ceilings labeled (est) in output.
Models
Matched by token against the model name vLLM reports at startup.
Model
Total params
Active params
MoE
Llama 4 Maverick
400B
17B
Yes
Llama 4 Scout
109B
17B
Yes
Llama 3 405B
405B
-
No
Llama 3 70B
70B
-
No
Llama 3 8B
8B
-
No
Llama 3.2 3B
3B
-
No
Llama 3.2 1B
1B
-
No
Nemotron 70B
70B
-
No
Nemotron 8B
8B
-
No
Muse Glimmer 30B
29.6B
27.8B (text)
No
Qwen3.8 27B
27.78B
27B (text)
No
Qwen3.6 27B
27B
-
No
Qwen3 235B
235B
22B
Yes
Qwen3 32B
32B
-
No
Qwen3 30B
30B
3B
Yes
Qwen3 14B
14B
-
No
Qwen3 8B
8B
-
No
Qwen3 4B
4B
-
No
Qwen3 1.7B
1.7B
-
No
Qwen3 0.6B
0.6B
-
No
Qwen2.5 72B
72B
-
No
Qwen2.5 32B
32B
-
No
Qwen2.5 14B
14B
-
No
Qwen2.5 7B
7B
-
No
Qwen2.5 3B
3B
-
No
Qwen2.5 1.5B
1.5B
-
No
Qwen2.5 0.5B
0.5B
-
No
DeepSeek 70B
70B
-
No
DeepSeek 7B
7B
-
No
DeepSeek R1 Distill Llama 70B
70B
-
No
DeepSeek R1 Distill Qwen 32B
32B
-
No
DeepSeek R1 Distill Qwen 14B
14B
-
No
DeepSeek R1 Distill Qwen 7B
7B
-
No
DeepSeek R1 Distill Llama 8B
8B
-
No
DeepSeek R1 Distill Qwen 1.5B
1.5B
-
No
Mistral Large 123B
123B
-
No
Mixtral 8x22B
141B
39B
Yes
Mixtral 8x7B
47B
13B
Yes
Mistral 7B
7B
-
No
Gemma 4 31B/27B
31B
-
No
Gemma 4 26B-A4B
25.2B
3.8B
Yes
Gemma 3 27B
27B
-
No
Gemma 3 12B
12B
-
No
Gemma 3 4B
4B
-
No
Gemma 3 1B
1B
-
No
Gemma 2 2B
2B
-
No
Gemma 27B
27B
-
No
Gemma 9B
9B
-
No
GLM 32B
32B
-
No
Phi-4 mini
3.8B
-
No
Phi 4 14B
14B
-
No
MoE models: active_param_count drives roofline ceilings; param_count (total) drives weight_gb and OOM headroom. Catalog includes models whose 4-bit weights fit the largest listed GPU (~288 GB). DeepSeek 671B / R1 / V3, Mistral Large 3 675B, Kimi K2, and GLM 744B are omitted. Qwen3 has no official 7B or 72B dense SKU. Muse Glimmer 30B is dense text plus a vision encoder: 29.6B total, 27.8B text stack on the decode roof. Qwen3.8 27B is hybrid DeltaNet + attention: 27.78B loaded (safetensors), 27B text stack on the decode roof. Qwen3.6 27B is the same hybrid class. The 2.4T Qwen3.8 Max is omitted. Gemma 4 27B is catalogued as 31B (Google's HF name vs param count). Unrecognized models skip roofline entirely. To add a model or GPU, open an issue or submit a PR to the catalog files in src/context/.
Profile measures behavior under load. Idle time has no waste to find. Ceilings are catalog-derived upper bounds, labeled (est). Missing data stays absent (None), never guessed.
Roofline: hardware ceilings
Baseline inputs
Input
Resolution
roofline_params
active_param_count if present, else param_count. Decode and prefill ceilings, efficiency %. Not ridge: ridge_batch_size is GPU peak FLOPS, peak BW, and weight dtype bits only (param count cancels in the weight-decode intensity model).
weight_params
param_count if present, else active_param_count. weight_gb and kv_headroom_gb (OOM check).
bytes_per_param (weights)
vLLM-reported quantization → QUANTIZATION/VLLM_QUANTIZATION env → vLLM-reported dtype → DTYPE/VLLM_DTYPE env → catalog default → bf16 (2 bytes). fp8 = 1, fp16/bf16 = 2, fp32 = 4. kv_cache_dtype sets KV element bytes only, never weight width. Source labeled in output when fallback is used.
tensor_parallel_size
--tensor-parallel-size CLI flag only in practice (TP >1 is refused at launch). Defaults to 1. Scales both ceilings and per-GPU KV headroom.
decode_ceiling_tps is TP-scaled (peak_bw × tp from the roofline above). The denominator is the absolute hardware ceiling, independent of current traffic. An idle server reads low, correctly. Requires generation_tokens_per_sec > 0. When measured decode beats the one-token-per-read roof (speculation suspected), efficiency, config-relative efficiency, and headroom are cleared: the scoreboard shows -, not a false %. Inside the estimate band without a speculation flag, values may still clamp; values derived from the estimated ceiling carry a tilde (~) in output.
Calibrated for decode-bound workloads. Prefill-heavy workloads (long prompts, short outputs) show artificially low efficiency %. The prefill ceiling is the relevant constraint there, not decode.
kv_headroom_gb: per-GPU after the 3.0 GB activation buffer; negative means current TP is insufficient. tpot_floor_ms and prefill_floor_ms: theoretical minimum latencies at ceiling; actual values above these indicate overhead.
Economics
Metric
Formula
tok / W
generation_tokens_per_sec / power_watts
J / token
power_watts / generation_tokens_per_sec
$ / 1M output tok
cost_per_hr × 1e6 / (generation_tokens_per_sec × 3600). Omitted when the turnover gate fails (generation_tokens_completed < mean running): not enough completions to trust the rate.
Cost priority: --cost-per-hour flag, then GPU catalog on-demand price, then absent. Catalog-derived $/1M output tok is labeled (est) in output; user-provided rates are not. Energy metrics (J/tok, tok/W) use energy-pair windows only (aligned GPU power and vLLM tok/s in the same window).
Closed-loop delta
Computed after every re-measure cycle. Each metric line shows before → after. A worse suffix marks material regressions only (not every negative delta).
When config drifted between windows (vLLM restart detected), the line reads Config changed. Baseline reset. and the efficiency arm of the delta is discounted; throughput alone decides improvement direction after a reset.
On regression: the degraded state becomes the new baseline and is pushed to the history stack. The loop continues through the normal apply/remeasure wait; there is no extra confirmation step beyond that.
Field
worse threshold
Throughput
Drop ≥5%
Efficiency (pp)
Drop ≥1.0pp
Cost/1M output tok
Rise >$0.01
J/tok
Rise >0.02
TTFT avg
Rise >5ms
TPOT avg
Rise >0.5ms
Per-window collection
Window size: 2s when --duration is 30s or less; 10s otherwise. Each window polls GPU (NVML, or AMD via libamdgpu_top) and vLLM in parallel at ~250ms intervals. These formulas produce one snapshot before multi-window aggregation.
Metric type
Formula
On failure
Histogram mean (TTFT, TPOT, prefill, queue, prompt tokens)
Δsum / Δcount first→last scrape; ×1000 for seconds-based latencies
Last-scrape cumulative mean when Δcount ≤ 0
Histogram p95 / p99
Delta buckets, then linear interpolation at q=0.95 or q=0.99. If target falls in +Inf bucket, returns last finite bucket's upper bound (clamped).
None. No cumulative fallback (stale percentile is worse than none)
Counter rate (tok/s, req/s, preemptions/s)
(last − first) / window_duration_secs
None on negative delta or zero duration. Zero delta is valid idle, not missing
Prefix cache hit rate
Δhits / Δqueries
None when Δqueries ≤ 0 (never divide by zero)
GPU util, power
Mean across window polls
Poll skipped if GPU field absent
VRAM, temp (current)
Last poll
Peaks (vram_peak_mb, temperature_peak_c, kv_cache_peak_perc): max across polls/scrapes
Each window produces one snapshot. Fields aggregate by type across windows. Active: server under real load. Evaluable: endpoint responded with valid data.
Merge per-window delta bucket vectors (sum counts at matching boundaries), recompute q=0.95 and q=0.99 via linear interpolation. Length or boundary mismatches (e.g. version skew) skip the window gracefully instead of aborting. Never average scalar percentile values. Scoreboard latency shows p95.
KV cache avg, prefix cache hit rate, prompt_tokens_mean
Evaluable
KV avg: time-weighted mean. Prefix hit rate: Σ Δhits / Σ Δqueries across all evaluable windows.
KV / VRAM / temp peaks
Evaluable
max(per-window peak, last evaluable landing value) so aggregate peak ≥ displayed current.
State gauges (VRAM used, temp, sm_clock)
Evaluable (last)
Last evaluable window's landing value.
Cumulative counters (total tokens, total reqs)
All (chronological last)
Chronologically last collected window. Idle tail included. Preserves true Prometheus server totals.
All windows non-evaluable
-
Chronologically last raw window returned in full.
Limitations
Every boundary, and where the math is approximate.
Product boundaries
One GPU.--tensor-parallel-size greater than 1 is refused. KV and weight sharding math is single-GPU only. Profile still tells you when a model needs tensor parallelism to fit at all.
vLLM only. The engine boundary is clean, but SGLang is not built yet.
Ceilings are uncalibrated. Published specifications overestimate. Every ceiling-derived number is marked (est) or with a tilde.
Overhead-bound is named, not measured. Profile can say the GPU is idling on CPU work but cannot quantify it.
Unknown GPU or model gets no ceiling. Profile reports Hardware ceiling unknown with the reason, rather than guessing.
No load, no answer. Idle windows are skipped. Drive load with vllm bench serve when you need traffic.
The mean is one collection. A diagnose run aggregates many 2s or 10s slices into one summary. Cyclic load averaged over that duration can look like a state it is not. Raise --duration so several cycles fit, up to 30m. Load that changes between iterations is not a duration problem. See Duration and traffic shape.
You apply the fix. Profile never changes your server.
Measurement gaps
Gap
Affects
In practice
MoE active parameters assume uniform routing
R4, baseline
If traffic heavily skews to specific experts, actual VRAM usage may differ from the baseline active_param_count approximation.
Sampling cliff / Temperature absent
R5
Sampling temperature is not exposed in vLLM metrics, so R5 cannot factor token diversity into concurrency saturation logic.
MoE weight uses catalog total param_count
R4, baseline
Runtime VRAM may be lower if not all experts are loaded
TP >1 not supported. Multi-GPU deployments are not supported today.
Baseline, R4, all rules
--tensor-parallel-size values above 1 are refused at launch. Profile still reports when a model needs TP to fit at all.
Efficiency % uses theoretical BW ceiling, not measured HBM
All rules
Can't distinguish BW bottleneck from scheduler under-feeding
FLOPs/s not measurable via NVML
R1
Prefill-heavy workloads show artificially low efficiency %
vllm_max_num_seqs absent in vLLM ≤0.18.0
R5
Requires -m flag
Chunked prefill: run can exceed max_num_seqs
R5
Samples above max + 0.5 are evaluated-false for the seat wall; R5 stays silent when the flag cannot be computed
GPU and model specs from catalog, not measured
Baseline
Output labels ceilings (est)
p99 absent on counter reset or zero-traffic window
Display
By design. Stale p99 is worse than none.
Design
The non-obvious choices and the reasoning behind them.
Option<T> over sentinels
Some(0.0) ≠ None. Every metric is optional. Missing data and zero are different. Treating them the same produces confident wrong answers.
Rules ranked by impact × confidence within the winning DAG layer. One primary signal per iteration. Five simultaneous recommendations produce paralysis, not action.
Delta, not snapshot
Each iteration reports what changed since the last: before → after values with worse labels on material regressions. A snapshot without temporal comparison is a status report, not a diagnosis.
Shared traffic gate
window_is_evaluable is one shared helper, used by all rules. Idle state has no waste to measure.
Agnostic Telemetry Interfaces
The src/collectors module abstracts raw Prometheus and GPU metrics into a standard RawSnapshot structure. The engine reasons on those snapshots. The collector boundary is the I/O seam; rules and the limiter still encode vLLM-shaped signals (max_num_seqs, KV gauges, preemption counters). A new inference framework needs a collector and rule/limiter work for its signals.
Deterministic Reasoning
The diagnostic engine is decoupled from CLI execution and network I/O. run_diagnose collects windows and builds the run-level aggregate; engine::build_report_for_diagnose takes that aggregate and produces the Report; output::stdout formats it. The loop runner (profiler::loop_runner) runs conditionally only when there are sufficient evaluable, non-idle windows. This separation means the reasoning layer is deterministic and historical telemetry can be replayed for exact regression testing.
250ms collection interval
Fast enough to catch GPU utilization transients that a 1s or 5s poll would average away. Slow enough that scrapes per window add no meaningful load to the vLLM endpoint or GPU telemetry path.
Peak over average for KV cache
KV cache usage is tracked as the maximum across all scrapes in a window, not only the last value. Near-full for rules is avg or peak at the pressure bar. A spike that causes preemptions can recover before the window ends; last scrape alone can bury it.
Borrowing in the engine hot path
The diagnostic pipeline uses AnalysisInput<'a> to borrow contexts across the engine without copying static context every window. Collectors still allocate per-window snapshots and histogram vectors. A profiler that eats its target's memory bandwidth is worse than no profiler.
Graceful degradation over Panics
When integrating parallel telemetry sources (vLLM and GPU), mismatches occur. Version skew can change bucket boundaries in histograms; instead of panicking, the engine skips the corrupted merge and degrades gracefully. One bad scrape window should never destroy the entire session's dataset.
Code Hygiene & Security
rustls only, no OpenSSL. Read-only GPU telemetry. No third-party telemetry leaving the machine; Profile only talks to the configured vLLM endpoint (and best-effort /v1/models, /info, and /server_info on that host). Missing endpoints or fields stay unknown; Profile does not invent config.
Every pull request and merge to main runs the same gate (see .github/workflows/build.yml and CONTRIBUTING.md). Socket reports supply-chain risk on the same events. Never mask a scanner with || true; allowlist real false positives in deny.toml.
cargo audit: RustSec Advisory Database against the dependency tree.
cargo deny: license, advisory, and ban policy via cargo deny check --all-features (deny.toml).
OSV-Scanner: fails on HIGH or CRITICAL vulnerabilities in Cargo.lock (CVSS v3 ≥ 7.0).
Semgrep SAST: static application security testing on the source.
Socket: supply-chain scanning on pull requests and merges to main (malware, typo-squatting, risky dependency behavior). GitHub App check: Socket Sec: Project Report.
cargo test --locked: unit tests for rules, physics math, and mock payloads.
The Inference Problem
Apr 19, 2026
TL;DR
Inference is 80-90% of your AI system's lifetime cost, not training.
Production servers bleed compute across batching gaps, KV cache pressure, prefill overhead, low throughput, and high latency.
The problems are fixable. Most teams can't see them.
You send a prompt. A GPU somewhere processes it and sends back a response, one token at a time. That loop is inference. It is also where most of the money goes.
80-90%of a production AI system's lifetime cost is inference, not training [1]
15×GPT-4 inference cost vs training cost. $2.3B in inference by end of 2024 vs ~$150M to train
While training is highly visible, inference constitutes the majority of expenditure.
An H100 rents for roughly $3/hr. When your GPU is underutilized, you get fewer tokens than the hardware can deliver. Same bill, lower output. That is over $1,000/month in cloud compute per GPU, wasted.
Common areas of compute waste include:
Throughput. A misconfigured vLLM server runs well below its hardware ceiling. When batching, memory, and scheduling are off, GPU utilization collapses. This represents a configuration bottleneck rather than a hardware limitation.
Latency. Real-time applications need time-to-first-token under 100ms. Most production setups don't get there without deliberate tuning.
KV cache pressure. Every token generated needs to remember everything before it. When that memory fills up, the system evicts. Throughput drops. Latency spikes. At 90% usage, you're already in trouble.
Batching. When batch size is 2 out of a possible 16, you're using 12% of the machine. The rest idles. You pay full price.
Prefill overhead. Before the first output token, the model processes your entire prompt. In workloads with 500+ token prompts, prefill alone dominates response time.
Observability. Most teams look at raw Prometheus metrics and guess. There is no standard path from "throughput is low" to "here is exactly why, here is the fix."
These are not edge cases. This is the normal state of a production inference server that has not been tuned.
Why This Matters Now
The standard story about early tech: Amazon lost money for years. Uber still does. Growth first, economics later. AI inference does not fit that story.
Amazon chose to lose money, a deliberate bet on demand. Companies running AI inference are burning money on hardware they already paid for, because it is sitting misconfigured. That is not a growth strategy. It is waste that shows up on your cloud compute bill this month.
Enterprises running production AI spend $50,000 to $500,000 per month on inference infrastructure. [2] Inference costs have dropped 280× since 2022, from $20 to $0.07 per million tokens. That gap between what hardware can do and what most teams extract from it is your cost problem today.
"The AI inferencing market will be much, much larger than the AI training market. People are running out of usable inference computing capacity." - Larry Ellison, Oracle earnings call [3]
The shortage is not GPUs. It is efficient use of the GPUs already running in production, including yours.
