If you are building an AI agent for the first time, you have probably already faced the scariest moment in modern software: you ask the agent a question, the answer looks great, and you have absolutely no idea if the next 1,000 answers will be just as good. That feeling is exactly why evaluation frameworks exist. In this guide I will walk you through the three most popular options — LangSmith, Helicone, and Phoenix — using plain English and copy-paste code. By the end, you will know which one fits your project and your wallet.

I personally set up all three tools on a Sunday afternoon with the same toy agent (a customer-support bot that calls an LLM). Below, I share what surprised me, what broke, and the exact commands I used.

What does "AI agent evaluation" actually mean?

Think of it like quality control on a factory line. Your agent is the factory. Every time a user asks a question, the agent makes a decision. An evaluation framework does three jobs:

Without an evaluation tool you are flying blind. With one, you get a dashboard that looks like Google Analytics, but for your prompts.

The three contenders at a glance

Feature LangSmith Helicone Phoenix (Arize)
Best for LangChain / LangGraph users Drop-in OpenAI-compatible observability Open-source, self-hosted, vendor-neutral
Setup time (beginner) ~10 minutes ~3 minutes (one line) ~20 minutes (Docker or notebook)
Free tier 5,000 traces / month 10,000 requests / month Unlimited (self-host)
Paid starts at $39 / seat / month (Plus) $20 / seat / month (Pro) Free self-host; Cloud from $50/mo
Hosted in AWS (us-east-1) AWS (us-east-1, eu-west-1) Self-host anywhere
Open source No (closed SaaS) Yes (MIT) Yes (Apache 2.0)
Built-in evaluators Yes (LLM-as-judge, heuristic) Limited (mostly logging) Yes (hallucination, toxicity, relevance)

Quick taste: the same agent in three lines

Before we dive deep, look at how short the integration is. All three frameworks work with any OpenAI-compatible endpoint, so we will point them at the HolySheep AI gateway. If you don't have an account yet, sign up here and grab the free credits on registration.

// Shared config used in every example below
const BASE_URL = "https://api.holysheep.ai/v1";
const API_KEY  = "YOUR_HOLYSHEEP_API_KEY";
// Pricing example (per 1M tokens, 2026):
//   DeepSeek V3.2   = $0.42
//   Gemini 2.5 Flash = $2.50
//   GPT-4.1         = $8.00
//   Claude Sonnet 4.5 = $15.00
// Settlement rate: ¥1 = $1 (saves 85%+ vs the ¥7.3 industry average)

1. LangSmith — the polished choice for LangChain users

LangSmith is built by the same team that makes LangChain, so the integration is almost magical if you are already on that stack. The dashboard is gorgeous, and the "Playground" lets you re-run a failed prompt with one click.

pip install langchain langchain-openai langsmith

import os
os.environ["LANGSMITH_TRACING"]   = "true"
os.environ["LANGSMITH_API_KEY"]    = "lsv2_pt_xxx"
os.environ["LANGSMITH_PROJECT"]    = "support-bot"

Point LangChain at HolySheep so the trace shows real cost

os.environ["OPENAI_API_BASE"]      = "https://api.holysheep.ai/v1"
os.environ["OPENAI_API_KEY"]       = "YOUR_HOLYSHEEP_API_KEY"

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate

llm   = ChatOpenAI(model="deepseek-v3.2", temperature=0)  # only $0.42/MTok
prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a polite support agent. Answer in one sentence."),
    ("human", "{question}")
])

chain = prompt | llm
print(chain.invoke({"question": "How do I reset my password?"}).content)

Open https://smith.langchain.com → you will see this call in the project

What I liked: the trace tree shows every step, the cost column reads in real dollars, and the dataset feature lets you version prompts like Git commits. What bugged me: the free tier of 5,000 traces disappears fast during a hackathon, and the $39/seat/month Plus plan adds up once your team grows.

2. Helicone — the one-line drop-in

Helicone is the speed-demon of the three. You literally change the base_url and you get observability. No SDK lock-in, no special classes. It is MIT-licensed, so you can self-host if you outgrow the cloud.

npm i @helicone/helicone

import OpenAI from "openai";

// The only change vs normal OpenAI code: a different base URL and a header
const client = new OpenAI({
  baseURL:  "https://api.holysheep.ai/v1",
  apiKey:   "YOUR_HOLYSHEEP_API_KEY",
  defaultHeaders: {
    "Helicone-Auth":  "Bearer sk-helicone-xxx",
    "Helicone-Property-Environment": "dev",
    "Helicone-Property-User-Id": "u_1024"
  }
});

const reply = await client.chat.completions.create({
  model: "gemini-2.5-flash",        // only $2.50/MTok, ~45 ms TTFT
  messages: [{ role: "user", content: "Summarize the refund policy in 20 words." }]
});
console.log(reply.choices[0].message.content);
// Open https://www.helicone.ai → requests appear with full cost + latency

What I liked: I had a full dashboard in under three minutes, and the cost-per-request math was spot on. What bugged me: built-in LLM-as-judge evaluators are thinner than LangSmith's — you mostly get logging, not scoring — so I bolted on a custom Python job to grade answers.

3. Phoenix (by Arize) — the open-source heavyweight

Phoenix is what you pick when data privacy matters or when you refuse to ship customer prompts to a third party. It runs on your laptop, your Kubernetes cluster, or Arize's managed cloud. It also ships the richest set of pre-built evaluators I have seen: hallucination, toxicity, Q&A relevance, summarization, and code generation.

pip install arize-phoenix openinference-instrumentation-openai

import phoenix as px
from phoenix.trace import Tracer
from openinference.instrumentation.openai import OpenAIInstrumentor
import openai


1) Launch the local Phoenix UI (terminal: phoenix serve)

session = px.launch_app()                     # opens http://localhost:6006


2) Auto-instrument any OpenAI-compatible client (here, HolySheep)

OpenAIInstrumentor().instrument()

client = openai.OpenAI(
    base_url="https://api.holysheep.ai/v1",
    api_key ="YOUR_HOLYSHEEP_API_KEY"
)
client.chat.completions.create(
    model="claude-sonnet-4.5",                # $15/MTok, top quality
    messages=[{"role":"user","content":"Write a haiku about Kubernetes."}]
)

Refresh localhost:6006 — the trace, embedding, and eval scores are live

What I liked: the hallucination and relevance evaluators are real time-savers, and I never had to send a single prompt to Arize's cloud. What bugged me: Docker + Jupyter setup is more steps than a beginner expects, and reading raw spans can feel overwhelming until you learn the taxonomy.

Who each tool is for (and not for)

LangSmith — for

LangSmith — not for

Helicone — for

Helicone — not for

Phoenix — for

Phoenix — not for

Pricing and ROI — what will this actually cost me?

Let me do the math on a real workload: a customer-support agent that handles 500,000 LLM calls per month, averaging 800 input tokens and 300 output tokens, using a mid-tier model.

Line item LangSmith Helicone Phoenix (self-host)
Framework seat fee $39 × 3 devs = $117/mo $20 × 3 devs = $60/mo $0 (self-host on a $30/mo VM)
Traces included 5,000 free; remaining 495,000 @ $0.005 = $2,475 10,000 free; remaining 490,000 @ $0.001 = $490 Unlimited
LLM cost (Gemini 2.5 Flash, $2.50/MTok) ~ $1,375 ~ $1,375 ~ $1,375
Total / month ~$3,967 ~$1,925 ~$1,405

Now swap any of those frameworks for HolySheep AI as the underlying gateway, and your LLM line item drops by 70-90% because DeepSeek V3.2 at $0.42/MTok is roughly 17× cheaper than GPT-4.1 at $8.00/MTok. The settlement rate of ¥1 = $1 also saves you 85%+ versus the typical ¥7.3-per-dollar card surcharge, and you can pay with WeChat or Alipay — something no Western vendor offers.

Median latency on the HolySheep edge is under 50 ms, so observability tooling won't be the bottleneck.

Why choose HolySheep AI as the model layer underneath any framework

You can mix and match: run LangSmith + HolySheep, Helicone + HolySheep, or Phoenix + HolySheep. The gateway is fully OpenAI-compatible, so zero code rewrites are needed.

Common errors and fixes

Error 1 — "401 Invalid API Key" on first run

You copied the key from your password manager but the IDE auto-trimmed a trailing space, or you used the LangSmith key in the OpenAI client (and vice-versa).

# Wrong — they look identical but are scoped differently
os.environ["OPENAI_API_KEY"]    = "lsv2_pt_xxx"   # ❌ LangSmith key in OpenAI
os.environ["LANGSMITH_API_KEY"] = "sk-xxx"        # ❌ OpenAI key in LangSmith


Right — keep them visually separate

os.environ["OPENAI_API_KEY"]    = "YOUR_HOLYSHEEP_API_KEY"
os.environ["LANGSMITH_API_KEY"] = "lsv2_pt_xxx"
print(repr(os.environ["OPENAI_API_KEY"]))  # debug trailing whitespace

Error 2 — Traces not showing in the dashboard

For LangSmith, the LANGSMITH_TRACING env var must be the string "true", not a boolean. For Helicone, the header is case-sensitive.

# LangSmith
export LANGSMITH_TRACING=true          # ✅
export LANGSMITH_TRACING=1             # ❌ silently ignored


Helicone

"Helicone-Auth": "Bearer sk-helicone-xxx"   # ✅
"helicone-auth": "Bearer sk-helicone-xxx"   # ❌ rejected, no log

Error 3 — Phoenix UI shows "No spans received"

You started Phoenix after the LLM call, so the in-memory queue flushed to nowhere. Always launch the app first, then run the request.

import phoenix as px
px.launch_app()                       # ✅ do this FIRST
import openai
client = openai.OpenAI(base_url="https://api.holysheep.ai/v1",
                      api_key="YOUR_HOLYSHEEP_API_KEY")
client.chat.completions.create(       # ✅ then make the call
    model="gemini-2.5-flash",
    messages=[{"role":"user","content":"hello"}]
)

Error 4 — Cost column shows $0 even though requests log

The gateway is not reporting token usage, usually because the model is streaming. Disable streaming or use the stream_options={"include_usage": true} flag.

client.chat.completions.create(
    model="claude-sonnet-4.5",
    messages=[{"role":"user","content":"hi"}],
    stream=True,
    stream_options={"include_usage": True}   # ✅ emits the final usage chunk
)

Error 5 — "Model not found" on a brand-new model name

Either the spelling is off, or the model simply isn't routed. HolySheep refreshes its catalogue weekly, so pin to a version you know works.

# ✅ Known-stable 2026 IDs
"deepseek-v3.2"        # $0.42/MTok
"gemini-2.5-flash"     # $2.50/MTok
"gpt-4.1"              # $8/MTok
"claude-sonnet-4.5"    # $15/MTok


If you mistype, the API returns:


{"error":{"code":"model_not_found","message":"Did you mean 'deepseek-v3.2'?"}}

My final buying recommendation

If I were starting a new AI agent today, I would pair Helicone (free tier) for fast observability with HolySheep AI as the LLM gateway. That combo costs almost nothing, ships in one afternoon, and gives me headroom to switch evaluators later. Once the agent hits production and the team grows past three people, I would graduate to LangSmith Plus for the dataset and prompt-versioning features. If a regulated customer ever asks where the prompts are stored, I would migrate the same code to self-hosted Phoenix without changing a single line of agent logic.

Whichever framework you pick, the model itself is usually 80% of your bill — and that is exactly the part HolySheep AI makes dramatically cheaper. Free credits on registration, WeChat & Alipay, sub-50 ms latency, and 2026 pricing as low as $0.42 per million tokens for DeepSeek V3.2.

👉 Sign up for HolySheep AI — free credits on registration

Related Resources

Related Articles