LangSmith - LLM Observability Platform

Development platform for debugging, evaluating, and monitoring language models and AI applications.

When to use LangSmith

Use LangSmith when:

• Debugging LLM application issues (prompts, chains, agents)
• Evaluating model outputs systematically against datasets
• Monitoring production LLM systems
• Building regression testing for AI features
• Analyzing latency, token usage, and costs
• Collaborating on prompt engineering

Key features:

• Tracing: Capture inputs, outputs, latency for all LLM calls
• Evaluation: Systematic testing with built-in and custom evaluators
• Datasets: Create test sets from production traces or manually
• Monitoring: Track metrics, errors, and costs in production
• Integrations: Works with OpenAI, Anthropic, LangChain, LlamaIndex

Use alternatives instead:

• Weights & Biases: Deep learning experiment tracking, model training
• MLflow: General ML lifecycle, model registry focus
• Arize/WhyLabs: ML monitoring, data drift detection

Quick start

Installation

pip install langsmith

# Set environment variables
export LANGSMITH_API_KEY="your-api-key"
export LANGSMITH_TRACING=true

Basic tracing with @traceable

from langsmith import traceable
from openai import OpenAI

client = OpenAI()

@traceable
def generate_response(prompt: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}]
    )
    return response.choices[0].message.content

# Automatically traced to LangSmith
result = generate_response("What is machine learning?")

OpenAI wrapper (automatic tracing)

from langsmith.wrappers import wrap_openai
from openai import OpenAI

# Wrap client for automatic tracing
client = wrap_openai(OpenAI())