GPU inference: test throughput inside a latency budget
A GPU inference benchmark is useful only when its output can meet the service being sold. A system that generates more tokens while users wait several seconds for the first response may support a different product from one that starts promptly. The research task is to measure successful useful output inside an explicit latency and quality budget, then identify how far that measurement can support a capacity claim.
Build a benchmark evidence sheet rather than rank chips by a single tokens-per-second number. Record the model, precision, workload, hardware, serving software, request arrival pattern and latency distribution together. A result applies to that configuration and test window; it is not automatically a revenue forecast or proof of equivalent performance on another customer’s prompts.
Locate the metrics in a primary source
NVIDIA’s LLM Inference Benchmarking: Fundamental Concepts explains queueing, prefill and generation under How LLM inference works. Its LLM inference metrics section and Figure 1 distinguish time to first token, inter-token latency, user token throughput and system token throughput. Read the article’s body and metric definitions, rather than treating a page summary as a substitute for the benchmark specification.
Time to first token, or TTFT, measures the wait before output begins. Inter-token latency concerns spacing during generation. System throughput aggregates work across concurrent requests, while a user’s rate describes that request’s experience. These objects have different denominators. A large aggregate rate can coexist with slow individual responses when concurrency increases.
Start with the service requirement. For this fictional example, requests must have TTFT no greater than one second and completion time no greater than twelve seconds. Every request has an identical two-hundred-token output target, a fixed input-length distribution and an unchanged quality test. These are invented service gates, not NVIDIA recommendations. Both latency conditions must hold for the same request.
Compare three measured configurations
Suppose one GPU is tested at concurrency one, eight and thirty-two. System output rates are respectively eighty, 480 and 900 tokens per second. Measured p95 TTFT values are 0.2, 0.8 and three seconds. All runs use the same output lengths, input sample, model version and measurement interval definition.
| Concurrency | System output tokens/second | p95 TTFT, seconds | Passes the run-level p95 TTFT gate? |
|---|---|---|---|
| 1 | 80 | 0.2 | Yes |
| 8 | 480 | 0.8 | Yes |
| 32 | 900 | 3.0 | No |
The thirty-two-request configuration has the highest bare throughput, but fails this particular run-level latency test. The eight-request configuration is the faster of the two configurations that pass that screen. That does not mean all requests in it meet the service gates. A percentile is a summary of a distribution, and separate percentile summaries do not reveal whether both conditions hold on the same observations.
This distinction changes the evidence requested from a vendor. Ask for timestamped request records, including submissions, first outputs, completions, failures and generated output counts. A chart containing average latency alone cannot reproduce the table’s p95 screen. Even a p95 chart cannot establish the joint success count without the relevant request-level records.
Rebuild useful output from the request ledger
For the concurrency-eight configuration, assume a separate fixed-window observation lasts five hundred seconds. Exactly one thousand requests enter the defined accounting cohort, and all are followed to their terminal status. Successful completions each contain two hundred output tokens. Of these requests, 930 meet both latency gates and the unchanged quality requirement; seventy fail at least one gate. The measurement design has no unassigned warm-up requests or carryover tokens.
Qualified output is 930 × 200 = 186,000 tokens. Qualified output rate is 186,000 ÷ 500 = 372 tokens per second. If all one thousand requests complete their target, total generated output is 200,000 tokens, or four hundred tokens per second. This fixed-window run is separate from the preceding illustrative 480-token benchmark result: different runs need their own denominators and cannot be silently combined.
The useful fraction is 93% of requests in this deliberately equal-length cohort. For unequal outputs, that percentage is not also the useful fraction of tokens. Count each qualifying request’s actual output, sum those counts and divide by the documented window. Keep requests failing quality separate from requests failing timing, while counting the union only once when calculating overall failure.
Do not multiply a system rate by concurrency again. The system total already includes concurrent work. Likewise, input tokens processed during prefill are not interchangeable with output tokens generated during decoding. If a reported rate includes both, request a separated series before comparing it with this worksheet’s output-only rate.
Separate measured capacity from an expansion scenario
Suppose eight independent replicas are planned and management uses a 70% planning factor. Applying that explicit assumption to the measured 372 gives 372 × 8 × 70% = 2,083.2 qualified output tokens per second. This is a capacity scenario, not an eight-GPU measurement. The factor does not prove that interconnects, the request router or shared storage will behave independently.
The scenario also assumes a comparable workload reaches every replica and the service gates remain unchanged. An average utilization factor cannot repair peak queueing that causes the latency objective to fail. Validate expansion with a new run that includes the production routing path, relevant arrival bursts and shared dependencies. Until then, show the single-replica observation and the expansion assumption in separate cells.
Test a boundary case with a different bottleneck
Now replace short prompts with a much longer input distribution while preserving the output target. Prefill work can increase and the first-token wait can change even if the hardware’s decoding rate appears stable. The old configuration’s TTFT result no longer answers the new service question. Reusing the previous output rate would conceal the changed workload.
Another boundary is quality. A lower-precision configuration may run faster but needs an unchanged task-specific quality evaluation before its output belongs in the same useful-capacity comparison. Merely using the same model family name does not establish equivalent responses. Record model checkpoint, precision, relevant serving settings and the quality acceptance method without inventing an accuracy claim.
Failures and retries introduce a further accounting problem. Retried requests consume hardware work but should not become additional successful customer tasks. Keep attempted requests, unique tasks, successful tasks and generated tokens as separate totals. If abandoned requests disappear from the latency dataset, the reported distribution may describe survivors rather than the service offered to all arrivals.
Use an executable verification sheet
| Input | Evidence to obtain | Reproduction check |
|---|---|---|
| Workload | Prompt-length and output-length records | Same distribution and tokenizer |
| Configuration | Model and serving manifest | Same checkpoint, precision and GPU count |
| Timing | Request timestamps and window definition | Queueing, warm-up and terminal status included |
| Quality | Fixed test and acceptance results | Same tasks and threshold across runs |
| Output | Per-request output and unique-task IDs | No input/output or retry double counting |
| Scaling | Replica run or explicit assumption | Observation separated from scenario |
When a sales slide and a log disagree, reconcile workload, percentile, successful-request population and test duration first. Do not average rates from incompatible runs. If timestamps or failed requests are unavailable, report only the narrower metric the evidence supports and mark the joint service-gate result unknown.
Update the worksheet when the model, input mix, quality threshold, scheduler, replica topology or service target changes. Preserve the earlier configuration so an improvement can be attributed to a specific change. The final research conclusion should state which workload and service budget the measured capacity serves, and which part still rests on an untested scaling assumption.
Audit the population behind a latency improvement
Before accepting a faster result, reconcile incoming requests with completed, failed, cancelled and still-running requests. A run that drops its slowest requests can show an improved completion percentile while serving fewer unique customer tasks. Keep the status of every arrival, and document how requests extending beyond the timing window are followed.
For variable output lengths, store both the requested maximum and actual generated length. Early stopping can improve apparent completion time while reducing output. That may be a useful product change, but it cannot be attributed solely to hardware speed. Use comparable task completion and quality criteria, or present the shorter-output configuration as a separate service.
Energy or cost comparisons require the same evidence boundary. Dividing a GPU’s rated power by a system-wide useful-output rate would mix a component estimate with an observed service total. Obtain measured energy for the defined serving system if making an energy-intensity claim. If only instance rental cost is known, describe the result as rental cost per qualified output under that run, not total business cost or gross margin. Keep utilisation and service-level failures visible when translating the capacity evidence into a commercial sensitivity.
FAQ
Does passing p95 TTFT mean every request meets the latency target?
No. It is a distribution summary and does not establish every request’s result or the joint completion-time gate.
Can eight GPUs be valued at eight times a one-GPU benchmark?
Only as a clearly stated scenario until the full serving arrangement is measured under comparable conditions.