Catch silent cost surges and broken agent loops before they ship.
Your agent returned the right answer, but took 5 extra steps and burned 3× the tokens. AgentDiff catches silent tool loops, cost surges, and execution drift in CI before your PR lands in production.
Automated PR checks · Blocks broken runs before code merges
Your unit tests are green.
Your agent is silently burning money.
Traditional assertions only check if the final output string matched. They are blind to execution drift — missing recursive tool loops, unverified prompt detours, and 3× cost surges before code merges.
Final output looks fine
The agent still delivered the expected summary, so standard assertion tests pass easily in CI.
3× redundant API calls
Behind the scenes, a prompt tweak caused the agent to loop 3 times over the customer database.
Instant merge block
AgentDiff detects the step surge in milliseconds and fails the PR before it ever costs you real money.
From git commit to merge gate.
No complex dashboards or cloud databases. AgentDiff operates as a pure, deterministic regression engine directly in your terminal and CI/CD pipeline.
Record Golden Baseline
Run your agent on a verified test dataset. AgentDiff records its execution graph, tool call sequences, and token budget into a committed JSON file in your repository.
Align & Diff on Every PR
When an engineer refactors a prompt or upgrades a model, AgentDiff aligns the candidate execution against the baseline in GitHub Actions without calling any paid LLM judges.
Block Regressions with Exit Code 1
If the PR agent enters an infinite retry loop, drifts off its path, or surges token cost, the build halts immediately and an automated root-cause comment is posted to the PR.
Built for engineering teams shipping agentic AI to production.
Deterministic gates designed to protect four things: your monthly token budget, user latency, graph consistency, and engineering confidence.
Token & Cost Surge Gating
Enforce strict safety budgets (e.g. max +10% token overhead). If a new prompt variant or model version burns more tokens than your baseline, AgentDiff fails the PR immediately.
Infinite Tool Loop Interception
Isolates cyclical execution loops where an agent calls the same endpoint with identical parameters without state progress — stopping runaway recursive loops before merge.
Silent Model Fork Detection
When you upgrade from GPT-4o to o3 or Gemini, AgentDiff mathematically aligns both execution graphs to verify whether the agent followed your golden path or hallucinated an unverified detour.
Sub-10ms Automated CI Runs
Unlike LLM-as-a-judge evaluators that cost $0.05 per test and take 15 seconds, AgentDiff runs air-gapped on raw AST graphs in single-digit milliseconds.
Instant root-cause diagnosis right in GitHub PRs.
Engineers never have to hunt through raw terminal logs or third-party web apps. AgentDiff posts an actionable diagnostic comment directly to the Pull Request review timeline.
refactor(prompt): parse customer records with new schema #42
main from feat/parse-schema⛔ AgentDiff · Trajectory Regression Check (Exit Code 1)
Candidate execution trajectory breached regression tolerances against golden baseline.
| Gate | Measured | Threshold | Verdict |
|---|---|---|---|
| TDI (Trajectory Divergence) | 0.4285 | ≤ 0.2500 | ❌ FAIL |
| Stagnant Tool Retries | 3 loops | 0 loops | ❌ FAIL |
| Token Budget Delta | +148.2% | ≤ 10.0% | ❌ FAIL |
execute_sql node repeated 3 consecutive times with identical parameters without state progress.
f8c21a9max_tdi = 0.25
max_consecutive_loops = 0
max_token_delta_percent = 10.0
Works with your existing agent stack. Zero code rewrite.
You shouldn’t have to re-architect your codebase to protect your agents. AgentDiff connects directly to your existing telemetry formats and framework runtimes.
LangGraph
State Checkpoint DAGsIngest native JSON trace dumps from LangGraph directly into AgentDiff's deterministic regression engine. Sub-5ms parsing speed, zero telemetry lock-in, and full DAG verification.
# Ingest native LangGraph Checkpoints with zero instrumentation
from agentdiff import load_trace, compare
baseline = load_trace("runs/langgraph_golden.json")
candidate = load_trace("runs/langgraph_pr.json")
# Compare execution paths across cycles & tool nodes
report = compare(baseline, candidate)
assert report.stagnant_loops == 0Where AgentDiff fits in your stack.
AgentDiff doesn’t replace observability platforms or LLM judges. It completes your modern AI stack by filling the critical missing layer: instant, deterministic regression gates in CI.
AgentDiff
Evaluates PR candidate runs against committed golden baselines in under 5ms. Blocks infinite loops, prompt drift, and token surges before code reaches production.
LLM-as-a-Judge
Evaluates nuances like politeness, formatting, and hallucination on offline test suites. High latency, non-deterministic scores, and API costs make it unsuited as a hard CI gate.
Observability (OTel / Langfuse)
Captures live production spans, distributed traces, and latency charts. Vital for debugging what happened after an outage, but cannot prevent broken PRs from deploying.
Everything you need to know.
Deterministic agent regression testing demystified. If you have any other questions, our engineering docs and GitHub discussions are always open.
Stop agent regressions before they merge.
Install in seconds. Compare candidate executions against golden baselines, block infinite tool loops, and protect your token budget on every PR.
- name: Run AgentDiff Gate
uses: agentdiff/action@v1
with:
baseline: tests/golden_baseline.json
max-divergence: 0.25