The evaluation layer that makes your AI agents self-improve.

Agents don't throw errors. Most failures are silent. Helix monitors your agent traces to find where they broke down and gives you the fix.

Join the alphaSee an example
Start for freeno credit card
the problem

Agents don't throw errors.
They just act.

Engineers are shipping value to users fast, but evals are very hard and painful to build and maintain. Teams don't have a way to know what issues their agent has before a customer catches them unless they spend a lot of time looking at traces.

01

Building evals takes weeks. Most become stale in weeks.

A proper eval set takes weeks of engineering time, goes stale within a sprint, and still only covers what you thought to test. So most teams ship on instinct and hope.

observationWe had ground truths for the happy path. But this week we've changed the schema and need to rebuild them.
02

Hundreds of wrong answers. Zero alerts.

Agents don't throw exceptions. They just act, confidently and incorrectly, across hundreds of conversations before anyone notices.

observationThe agent had been issuing wrong refunds for two days. An engineer caught it while debugging something else.
03

Every change ships into uncertainty.

Agents are probabilistic. When you fix one thing, you have no way of knowing if something else quietly broke. There is no coverage to run against, no signal until a customer tells you.

observationPushed a fix Monday. Found out Thursday it broke the other flow. No way to catch it earlier.
The solution

Install Helix.
Every trace improves your agent, forever.

Helix wraps your agent to watch production behavior, evaluates it against your code's intent, proposes code-level fixes, and validates them before you ship.

~/project
helix · sources
$ npm install @helix/sdk
added 1 package in 0.8s
reading agent intent from codebase
// agents/support.ts
import { helix } from '@helix/sdk'
const agent = helix.wrap(myAgent)
LangSmith
connected
LangfuseLangfuse
connected
OpenTelemetryOpenTelemetry
connected
4,821 traces ingested
01

Install

Connect Helix to your traces. It reads the code to learn what each agent should do.

02

Judge every step

An AI judge evaluates every reasoning step in a production trace, not just the final output, and flags silent failures and intent mismatches against your code.

helix.app / org_atlas/traces/trace_8a2f1cjudge · running
support-agent · run #418claude-sonnet-4-6 · 5 steps
01user_message
intent understood
02llm_reason
reasoning valid
03tool_call
correct tool
04tool_result
data retrieved
05respondhallucination
4 ✓ · 1 ✗ · T+11.4sreplay
helix · judge logT+11.4s
step 01 user_messagepass

intent classified · refund · order #4421 · confidence 0.94

step 02 llm_reasonpass

chose to retrieve policy before responding · matches code intent

step 03 tool_callpass

get_refund_policy("4421") · correct tool · args valid

step 04 tool_resultpass

retrieved · 0 docs (policy unset for this order type)

step 05 respondsilent failure

policy fabricated · cited a 30-day window not in retrieved docs

proposed fix · agents/support/refund.ts:42
+ if(!docs.length) escalate()

// guard the empty-docs path before respond()

replayed1,247 tracesaccuracy87.4% → 94.2%delta+6.8% ▲
judge · every step · not just the final output
agents/support/respond.ts+3 −1
28 async function handleMessage(ctx: Context) {
29 const query = ctx.message.text;
30 const prompt = buildPrompt(ctx);
31− return respond(await llm.generate(prompt));
31+ const docs = await kb.search(query);
32+ if (!docs.length) return escalate();
33+ return respond(await llm.generate(prompt, docs));
34 }
helix proposes·validated on 1,247 traces
03

Propose the fix

On every deviation, Helix proactively suggests a code-level change. It lands in your repo as a diff with the failing trace attached.

04

Validate, then ship

Helix replays the proposed fix in a sandbox against your trace history and surfaces an accuracy delta before merge.

fix/support-agent-escalation+3 −1
helixcommented just nowsafe to merge

Replayed fix against 1,247 production traces. Accuracy improved from 87.4% to 94.2% with 0 regressions detected.

+6.8% accuracy·0 regressions·1,247 traces replayed
See more:Trace breakdown →Replay report →Diff →
validate · then ship · replay delta posted to the pr
Example

See the silent failure.
Ship the validated fix.

A support agent was asked why users were being logged out across devices. The knowledge base had no document covering it. Instead of saying it didn't know, the agent fabricated a policy. Standard evals scored it highly. Helix flagged it.

Based on a true storyCursor AI support bot hallucinated its own company policy - The Register
trace_8a2f1c
1 issue
TraceAI fabricates logout policy user cancels subscription
1user_message0.1s“I keep getting logged out whenever I switch between my laptop and desktop.”
2llm_reason0.8s
3tool_call0.3s
4tool_result
5tool_call1.2s
6tool_resultretrieval miss{ results: [], matched: 0 }
7llm_reasoninference gap1.1s
8respondhallucinationThe logout behavior is expected – our policy limits sessions to one device.
9log_event0.1s
10complete
Helix analysishigh

search_knowledge_base returned 0 results. No policy document covers multi-device sessions. Instead of escalating, the agent inferred from similar SaaS tools and responded with a fabricated one-device limit which does not exist. The user cancelled their subscription.

// step 8 · confidence 0.97 · 23 similar traces · 4 led to churn
agents/support/respond.ts+3 −1
31−return respond(await llm.generate(prompt));
31+const docs = await kb.search(query);
32+if (!docs.length) return escalate();
33+return respond(await llm.generate(prompt, docs));
replay sandbox1,247 traces
accuracy87.4%94.2%
delta+6.8%
regressions0
Ship fix & close loop
Why Helix

Other tools log agent activity
for humans to review. Helix fixes it.

Observability platforms, eval frameworks, LLM-as-judge models: every existing tool gives engineers better dashboards for reading traces. Helix replaces the human in the loop. It judges every step, proposes the fix, and validates it before you ship.

capability
observability + evals
llm-as-judge
helix
Trace ingestionCapture every step
yes
partial
yes
Step-level intent judgmentCatch silent failures inside the trace
no
yes
native
Code-level fix proposalsProactively suggests a code-level change
no
no
yes
Works without ground truth labelsNo labeling pipeline needed
no
partial
yes
Framework agnosticWorks across every stack
partial
partial
yes
Product-level contextLooks beyond the agent at the full user session
no
no
yes
No new pipeline

Helix plugs into the stack you already have.

LLM SDKs
OpenAI SDKOpenAI SDK
InstructorInstructor
Anthropic SDKAnthropic SDK
Azure OpenAIAzure OpenAI
Google GenAIGoogle GenAI
Vertex AIVertex AI
AWS BedrockAWS Bedrock
CohereCohere
LiteLLMLiteLLM
GroqGroq
MistralMistral
OllamaOllama
WatsonxWatsonx
Together AITogether AI
Aleph AlphaAleph Alpha
HuggingFaceHuggingFace
ReplicateReplicate
SageMakerSageMaker
Ruby LLMRuby LLM
Frameworks & Observability
OpenAI AgentsOpenAI Agents
Claude Agent SDKClaude Agent SDK
LangChainLangChain
LangGraphLangGraph
LangflowLangflow
LlamaIndexLlamaIndex
CrewAICrewAI
AutoGenAutoGen
DSPyDSPy
Pydantic AIPydantic AI
Google ADKGoogle ADK
SmolagentsSmolagents
Strands AgentsStrands Agents
GuardrailsGuardrails
HaystackHaystack
AgnoAgno
MCPMCP
BeeAIBeeAI
PipecatPipecat
MastraMastra
SuperagentSuperagent
BAMLBAML
Vercel AIVercel AI
LangfuseLangfuse
HeliconeHelicone
Arize PhoenixArize Phoenix
BraintrustBraintrust
OpenTelemetryOpenTelemetry
Alpha

Make your agents
self-improve.

Limited alpha spots. Join now and we'll reach out when your stack is ready to plug in.

Join the alphaTalk to a founder