ModelRefs / Load Testing (LLMs) — AI Glossary
Load Testing (LLMs) — AI Glossary
Simulating high-volume traffic against an LLM serving stack to measure throughput, latency distribution, and failure modes under stress.
Overview
LLM load testing tools: Locust (Python-based, customizable), k6 (JS, cloud-native), Artillery, and specialized tools like llmperf (OpenAI) and LiteLLM load test. Key metrics: requests/second capacity, P50/P95/P99 TTFT, tokens/second per user, and error rate under concurrency. Essential before launching high-traffic LLM features.
Reference details
| Topic | ecosystem |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Load Testing (LLMs) — AI Glossary.
Frequently asked questions
What is Load Testing (LLMs)?
Simulating high-volume traffic against an LLM serving stack to measure throughput, latency distribution, and failure modes under stress.
What concepts are related to Load Testing (LLMs)?
Closely related concepts include p99 latency, tokens per second, throughput.