AI Gateway — AI Glossary

Reference details

Related terms

• Model Gateway (LLM Gateway)
• Model Routing
• Rate Limiting

Example: What breaks without one

Three services each hold their own OpenAI key. Someone needs per-team spend, so every service gets patched. Then a provider outage needs a failover, so every service is patched again. A gateway makes those one change in one place, because the application only ever talks to the gateway.

Commonly confused with

A gateway is not a router. Routing — choosing which model serves a request — is one feature a gateway may offer. The gateway itself is the proxy layer: one API surface, with auth, logging, rate limits, caching and failover behind it.

When to use it

Reach for it when:

• More than one service or team calls model providers
• You need per-team cost attribution, quotas or a kill switch
• Provider failover has to work without redeploying applications

Reach for something else when:

• One service, one provider — it is infrastructure with nothing to manage
• Latency is critical and you cannot absorb an extra network hop
• You have not planned for it being a single point of failure

Referenced by

This term is used by the following ModelRefs references:

• byo model gateway stack

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to AI Gateway — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path