LLM Evaluation Platforms Converge on OpenTelemetry — And the Real Differentiators Have Shifted
Seven major LLM evaluation and observability platforms have converged on OpenTelemetry's GenAI Semantic Conventions as their telemetry backbone. The competitive battle has shifted to agent trajectory evaluation, eval-driven CI/CD workflows, and deployment model flexibility.
Editorial artwork pending
Most teams shipping LLM-powered applications in 2026 face the same quiet problem: they cannot tell you, with any confidence, whether Tuesday's prompt change made their system better or worse. They monitor uptime, they track latency, they might even log token costs. But systematic evaluation of output quality remains, for the majority, an afterthought bolted onto production systems months after launch.
The market has noticed. LLM observability and evaluation platforms now constitute a $2.69 billion category, projected to reach $9.26 billion by 2030 at a 36.2% compound annual growth rate, according to recent industry analysis. Gartner estimates that by 2028, LLM observability investments will account for 50% of generative AI deployments, up from 15% in early 2026. Seven major platforms — Arize Phoenix, Langfuse, LangSmith, Braintrust, Weights & Biases Weave, Comet Opik, and Pydantic Logfire — now compete across overlapping feature sets, each with a distinct architectural bet on how evaluation should work.
And yet, beneath the competitive noise, something more structurally significant is happening: convergence. Nearly every platform in the category has adopted OpenTelemetry's GenAI Semantic Conventions as its telemetry backbone, and that single technical decision is reshaping what switching costs and vendor lock-in mean for AI infrastructure teams.
The OpenTelemetry Standard as Common Ground
OpenTelemetry's gen_ai.* span attributes, now at version 1.37 and above, have become the vendor-neutral contract for LLM telemetry. Arize AX and Phoenix built their instrumentation natively on OTel from the start. Langfuse's March 2026 observations-centric data model shipped with an OTel processor and @observe decorator pattern. Braintrust and Comet Opik expose OTLP endpoints for both cloud and self-hosted deployments. Even LangSmith, long anchored to LangChain's proprietary tracing, now supports OpenTelemetry ingestion alongside its native integration.
The practical consequence is portability. A team instrumenting their application with standard OTel spans can, in principle, route that telemetry to any platform in the category without re-instrumenting their codebase. In practice, the depth of platform-specific features — trajectory-level agent evaluation, eval-driven CI/CD gates, prompt optimization algorithms — still creates meaningful differentiation. The days of choosing a platform and accepting permanent instrumentation lock-in are ending.
Three Architectural Bets
The seven platforms cluster into three strategic positions, each reflecting a different theory about where value accrues in the evaluation workflow.
The open-source self-hosted cluster — Arize Phoenix (Elastic License 2.0), Langfuse (MIT core), and Comet Opik (Apache 2.0) — bets that teams want full data sovereignty and zero vendor dependency. Phoenix has crossed two million monthly downloads, establishing itself as the default for teams running evaluation infrastructure on their own hardware. Langfuse's v4 architecture, rebuilt on ClickHouse, reportedly delivers dashboard performance up to 165 times faster than its predecessor, addressing the scale ceiling that historically pushed self-hosted users toward commercial alternatives. Opik differentiates with prompt and tool optimization algorithms and an MCP server that lets agents query their own observability data — a recursive capability that hints at where autonomous AI systems are heading.
The framework-native position belongs primarily to LangSmith, which remains the default instrumentation layer for LangChain 1.0 and LangGraph 1.0 applications. Its LangSmith Engine clusters production failures into prioritized issues and traces root causes through both the trace graph and application code. For teams already committed to the LangChain ecosystem, the integration overhead approaches zero. The trade-off is equally clear: LangSmith's deepest capabilities require LangChain's abstractions, and teams running custom orchestration or alternative frameworks encounter more friction.
The eval-first commercial cluster — Braintrust and Arize AX — centers evaluation not as a monitoring add-on but as the primary development workflow. Braintrust's architecture connects production traces directly to regression test suites, and its Loop AI agent analyzes trace data to suggest prompt improvements automatically. The company secured one of 2026's largest funding rounds in the AI evaluation category, a Series B that signals investor confidence in evaluation-as-workflow over evaluation-as-observability. Arize AX extends this approach with Alyx, an AI agent for trace investigation and evaluator development, and introduces audio evaluation capabilities for voice-based AI applications — a frontier that most competitors have not yet addressed.
Agent Evaluation: The Frontier That Separates Platforms
The shift from single-call LLM applications to multi-step agentic systems has exposed a capability gap across the category. Evaluating whether a chatbot answered a question correctly is a fundamentally different problem from evaluating whether an agent navigated a ten-step tool-calling workflow and arrived at the right outcome through a reasonable trajectory.
Arize AX, Phoenix, LangSmith, and Braintrust now support trajectory-level evaluation — scoring not just the final output but the sequence of reasoning steps, tool invocations, and intermediate decisions an agent took. Langfuse, W&B Weave, and Opik offer partial trajectory support, with agent graph visualization and annotation workflows that capture session-level behavior without full trajectory scoring.
Together, these capabilities represent the evaluation stack's response to the agentic turn in AI development. Teams building autonomous systems — agents that browse the web, write code, manage infrastructure, or coordinate with other agents — need evaluation frameworks that can distinguish between an agent that reached the right answer through sound reasoning and one that stumbled into correctness through a fragile path that will fail on the next input.
What This Means for Engineering Teams
In practice, the choice between platforms now reduces to three questions. First, deployment model: does the organization require full data sovereignty (Phoenix, Langfuse, Opik), tolerate a managed cloud with hybrid options (Braintrust, LangSmith Enterprise, W&B), or prefer fully managed SaaS (Arize AX, Datadog LLM Observability)? Second, framework coupling: is the team committed to LangChain and LangGraph (LangSmith), to Pydantic AI (Logfire), or framework-agnostic (most others)? Third, evaluation maturity: does the team need basic tracing and cost monitoring today, or is it ready for eval-driven CI/CD with automated regression testing and trajectory analysis?
The market's convergence on OpenTelemetry means the third question — evaluation maturity — should drive the decision, not the first two. Infrastructure portability is real. A team that starts with Langfuse for self-hosted tracing can migrate to Braintrust for eval-driven development without re-instrumenting their application, because both consume the same OTel spans. The evaluation workflow, not the data pipeline, is where platforms now earn their position in the stack.
For engineering leaders evaluating this category, one heuristic cuts through the noise: treat OTel compatibility as a hard requirement, then choose the platform whose evaluation workflow matches your team's current maturity and ambition. The telemetry is portable. The workflow habits are not.
