Profile

Physics-aware optimization loop for vLLM inference servers.

Install

# Linux x86_64
wget https://github.com/jungledesh/profile/releases/latest/download/profile
chmod +x profile
./profile diagnose --url http://localhost:8000/metrics

# Rust toolchain
cargo install profile-vllm

Usage

profile diagnose --url http://HOST:8000/metrics
profile diagnose --url http://HOST:8000/metrics --duration 2m
profile diagnose --url http://HOST:8000/metrics -m 256

What profile detects

Catalog

GPU specs, model parameters, and cloud prices resolved once at startup.

GPUs

FP32 TFLOPs and memory bandwidth feed the roofline math. Prices are market-average estimates as of 2026-06. Use --cost-per-hour for exact rates.

Models

Matched by token against the model name vLLM reports at startup.

Math

The formulas Profile runs on collected data.

Roofline: hardware ceilings

Baseline inputs

Two ceilings, two binding constraints. LLM serving is memory-bandwidth-bound at decode and compute-bound at prefill.

peak_bw_gbps, peak_flops, and params come from catalogs, not live measurement. All ceiling outputs labeled (est).

Efficiency %

aggregate_ceiling = decode_ceiling_tps × num_requests_running
efficiency_pct    = actual_tps / aggregate_ceiling × 100

decode_ceiling_tps is TP-scaled (peak_bw × tp from the roofline above). Requires num_requests_running > 0 and generation_tokens_per_sec > 0. Not shown if actual_tps > aggregate_ceiling: that indicates a catalog or measurement mismatch.

Weight footprint and headroom

weight_gb        = weight_params × bytes_per_param / 1e9
kv_headroom_gb   = vram_total_gb − (weight_gb / tp)
headroom_pct     = 100 − min(efficiency_pct, 100)
tpot_floor_ms    = 1000 / decode_ceiling_tps
prefill_floor_ms = 1000 / prefill_ceiling_tps

kv_headroom_gb: per-GPU; each GPU holds weight_gb/N with TP=N. Negative means current TP is insufficient. headroom_pct: unused fraction of the decode ceiling. tpot_floor_ms and prefill_floor_ms: theoretical minimum latencies at ceiling; actual values above these indicate overhead.

Economics

tokens_per_hr = generation_tokens_per_sec × 3600. You pay cost_per_hr for that many tokens; scale to 1M.

Cost priority: --cost-per-hour flag, then GPU catalog on-demand price, then absent. Catalog-derived $/1M tok is labeled (est) in output; user-provided rates are not.

Closed-loop delta

throughput_delta_pct = (after − before) / before × 100
efficiency_delta_pp  = efficiency_pct_after − efficiency_pct_before

Computed after every re-measure cycle. Direction uses efficiency_delta_pp, not raw throughput. Dividing by num_running (time-weighted mean across the window) cancels traffic-induced concurrency changes; what remains is per-request hardware utilization.

Latency veto: if efficiency says Better but ttft_p99_delta_pct > +20%, direction is Plateau. Downgrades Better only; does not affect Plateau or Worse. Skipped on the throughput fallback path.

On Worse: loop pauses. Operator chooses [r] revert or [c] continue. On continue, the degraded state becomes the new baseline before the next iteration. On revert, baseline is unchanged.

All thresholds are provisional constants. Latency, cost, and p99 before/after are tracked alongside direction for display.

Per-window collection (~2s)

Each window polls at ~250ms intervals. GPU (NVML) and vLLM are collected in parallel threads on each poll. These formulas produce one snapshot before multi-window aggregation.

Collection cadence and edge cases: Data.

Aggregation: combining windows

Each ~2s window produces one snapshot. Fields aggregate by type across windows. Active: server under real load (requests running, KV or GPU util above threshold). Evaluable: endpoint responded with valid data.

Rules

Five rules. Each fires when a specific bottleneck is confirmed under load.

R1: Under-batching

Prefill saturation uses histogram window mass (Δsum / window_secs), not the roofline ceiling. A window with more than 40% of its duration spent in prefill, or more than 4 absolute seconds, is not under-batched.

When the prefill gate suppresses R1, verbose mode (-v) shows why: Under-batching: not triggered (prefill saturated at 55%). Without -v, the rule appears as not triggered with no reason.

Fix: Raise client concurrency.

R2: KV cache pressure

R2 has two sub-variants. Both surface under the same [!] KV Cache Pressure header. The main path suppresses the backlog path when both fire.

Main path

VRAM ≥78% is shown in output as corroborating evidence when present, but is not a required condition. R2 fires on KV usage or preemptions alone.

Fix: Reduce --max-num-seqs or --max-model-len.

Admission backlog path

Fires when the scheduler is holding requests in queue to protect KV memory, before KV usage crosses the 88% threshold. The fix is to expand the KV pool, not reduce concurrency.

Fix: Raise --gpu-memory-utilization if VRAM headroom exists; switch to fp8 KV cache; or reduce --max-model-len.

R3: Low prefix reuse

Fix: If prefix caching is disabled: enable --enable-prefix-caching, then restructure prompts. If already enabled: move shared instructions to the start, standardize templates, avoid unique tokens at the beginning.

R4: OOM risk

Fires regardless of traffic. Configuration fact, not runtime observation.

The 3 GB buffer (ACTIVATION_KV_BUFFER_GB) reserves headroom for activations and initial KV blocks. R4 fires when kv_headroom_gb < 0 (weight_gb / tp > per-GPU VRAM). min_tp uses gpu_memory_utilization (default 0.90) for the fix recommendation.

Fix: Profile computes required TP degree: "Set --tensor-parallel-size N".

R5: Concurrency saturation

Fix: Raise --max-num-seqs if KV peak <80%. If KV is at or above 80%, raising the cap will cause thrashing; add a replica or lower --max-model-len instead.

Confidence

Per-rule, not a unified formula. Each rule sets confidence based on what signals it has.

Rule significance gate

rule_is_significant = fired ≥ 3  AND  fired / total_evaluable_windows ≥ 25%

A rule that fired once is noise. Profile only surfaces a recommendation when the signal is persistent: at least 3 windows and at least 25% of all evaluable windows in the run.

R2 main path exception. Critical KV conditions bypass the standard gate early:

The admission backlog path and all other rules use only the standard gate.

Ranking

score = impact × confidence  // impact: 1–5, confidence: 0.0–1.0

Rules sorted by score, highest first. Grouped by root cause into IssueGroup. One primary recommendation surfaced per group: the rest are supporting evidence, not separate alerts.

Data

What, when, and how we collect.

Cadence

GPU and vLLM scraped in parallel, 250ms apart. One window is ~2 seconds, ~9 polls. Each collector timestamps on completion. If the two timestamps diverge by more than 1 second, rules that cross-reference both sources are skipped.

Within each window, different metric types are treated differently:

Evaluable window gate

Two-tier gate. Structural validity and active traffic are separate checks.

Tier 1: structural. Did the endpoint respond and does the window have a valid duration? All rules use this check.

window_duration_secs  is present and positive
num_requests_running  is Some (even if zero)

running = 0 is valid data: the server responded and reported no active requests. None means the endpoint did not respond.

Tier 2: active. Was the server doing real work? Aggregated means use this check. Throughput, efficiency %, GPU utilization, and latency averages are computed over active windows only.

running_reqs > 0
AND (kv_cache_pct > 30%  OR  gpu_util_pct > 20%)

// both absent: running_reqs > 0 alone
// one absent:  evaluate the other arm only

kv_cache_pct catches decode and sustained load. gpu_util_pct catches prefill bursts where the KV cache is still near-empty. Idle windows pass tier 1 but not tier 2 and are excluded from the averages.

Multi-window aggregation, which window set applies to which field, is in Math.

Source 1: NVML

NVIDIA Management Library. Direct access to GPU hardware state, polled at 250ms intervals.

Source 2: vLLM Prometheus

Scraped from /metrics at ~250ms intervals, ~9 times per window. See vLLM docs for the full metric reference.

Catalog-derived fields

Looked up once at startup. See Catalog for supported GPUs and models.

Fallbacks

Limitations (wip**)

Where the math is approximate and why.

Design (wip**)

The non-obvious choices and the reasoning behind them.

Option<T> over sentinels

Some(0.0) ≠ None. Every metric is optional. Missing data and zero are different. Treating them the same produces confident wrong answers.

Physics ceiling as a range

CeilingEstimate { lower, expected, upper }. Catalog-derived specs carry uncertainty. Output labels ceilings (est).

One recommendation per iteration

Rules grouped by root cause. One primary signal per group. Five simultaneous recommendations produce paralysis, not action.

Delta, not snapshot

Each iteration reports what changed since the last: direction, magnitude, likely cause. A snapshot without temporal comparison is a status report, not a diagnosis.

Shared traffic gate

window_is_evaluable is one function, called once, used by all rules. Idle state has no waste to measure.

Engine never imports CLI

The reasoning layer is deterministic and testable in isolation. Adding a new interface (SGLang, API) doesn't touch the engine.

250ms collection interval

Fast enough to catch GPU utilization transients that a 1s or 5s poll would average away. Slow enough that scrapes per window add no meaningful load to the vLLM endpoint or NVML.

Peak over average for KV cache

KV cache usage is tracked as the maximum across all scrapes in a window, not the last value. A spike that causes preemptions and latency degradation can recover before the window ends. The last scrape reports a safe number while the damage already happened. The peak is the signal. The average buries it.

Zero allocation in the hot loop

AnalysisInput<'a> borrows both contexts. No clones per 250ms window. A profiler that eats its target's resources is worse than no profiler.

Security

rustls only, no OpenSSL. Read-only NVML. No external API calls. No telemetry. cargo audit clean before every release tag.

The Inference Problem

Apr 19, 2026

You send a prompt. A GPU somewhere processes it and sends back a response, one token at a time. That loop is inference. It is also where most of the money goes.

Training gets the headlines. Inference pays the bills.

An H100 rents for roughly $3/hr. When your GPU is underutilized, you get fewer tokens than the hardware can deliver. Same bill, lower output. That is over $1,000/month in cloud compute per GPU, wasted.

Most teams are bad at it. The waste is in:

Throughput. A misconfigured vLLM server runs well below its hardware ceiling. When batching, memory, and scheduling are off, GPU utilization collapses. The hardware is not the problem. The configuration is.

Latency. Real-time applications need time-to-first-token under 100ms. Most production setups don't get there without deliberate tuning.

KV cache pressure. Every token generated needs to remember everything before it. When that memory fills up, the system evicts. Throughput drops. Latency spikes. At 90% usage, you're already in trouble.

Batching. When batch size is 2 out of a possible 16, you're using 12% of the machine. The rest idles. You pay full price.

Prefill overhead. Before the first output token, the model processes your entire prompt. In workloads with 500+ token prompts, prefill alone dominates response time.

Observability. Most teams look at raw Prometheus metrics and guess. There is no standard path from "throughput is low" to "here is exactly why, here is the fix."

These are not edge cases. This is the normal state of a production inference server that has not been tuned.

Why This Matters Now

The standard story about early tech: Amazon lost money for years. Uber still does. Growth first, economics later. AI inference does not fit that story.

Amazon chose to lose money, a deliberate bet on demand. Companies running AI inference are burning money on hardware they already paid for, because it is sitting misconfigured. That is not a growth strategy. It is waste that shows up on your cloud compute bill this month.

Enterprises running production AI spend $50,000 to $500,000 per month on inference infrastructure. [2] Inference costs have dropped 280× since 2022, from $20 to $0.07 per million tokens. That gap between what hardware can do and what most teams extract from it is your cost problem today.

"The AI inferencing market will be much, much larger than the AI training market. People are running out of usable inference computing capacity." — Larry Ellison, Oracle earnings call [3]

The shortage is not GPUs. It is efficient use of the GPUs already running in production, including yours.