ModelRefs / LangSmith — AI Glossary

LangSmith — AI Glossary

LangChain's observability and evaluation platform for LLM applications, providing tracing, testing, and dataset management.

Overview

LangSmith captures full execution traces of LangChain (and non-LangChain) LLM apps: every LLM call, tool invocation, retrieval, and intermediate step. Supports: dataset creation from production traces, prompt playground, evaluation runs with custom evaluators, and regression testing across model versions. The primary observability tool for LangChain-based apps.

Reference details

Topic	infrastructure
Last reviewed	2026-06-24

Example: The trace is the artifact, the dataset is the payoff

A user reports a bad answer. The trace shows the run as a tree: the retrieval span and what it returned, the rendered prompt actually sent, the tool calls with their arguments, and the final generation with token counts. That localises the fault to one span. The second use matters more over time — a trace that exposed a failure can be promoted into an evaluation dataset, so the same input is checked on every future change. Production traffic becomes the regression suite, which is the only source of test cases that reflects what users actually send.

Commonly confused with

LangSmith is observability and evaluation tooling, not an agent framework — it traces applications whether or not they are built with LangChain. Compare with Langfuse and with generic OpenTelemetry backends: the LLM-aware part is that spans carry prompts, token counts and tool arguments, which is also what makes them subject to the same redaction rules as any store of user text.

When to use it

Reach for it when:

Multi-step chains and agents, where a single log line cannot localise a fault
Building evaluation datasets from real failures rather than imagined inputs
Comparing prompt or model versions against a fixed dataset before shipping

Reach for something else when:

Single-call applications, where request logging is sufficient
Sending unredacted user content to a third-party trace store without checking your data agreements
As the only evaluation: a trace shows what happened, not whether it was correct

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to LangSmith — AI Glossary.

Frequently asked questions

What is LangSmith?