How to Tell If Your AI Agent Is Actually Working

You can tell whether your AI agent is actually working by checking five things: whether you can see every step of a real production run (a trace), whether an automated judgment scores answers against the evidence the agent retrieved (evals), whether cost and latency are attributed per run rather than per invoice, whether you can replay a bad run to see what the agent saw, and whether anyone would be alerted if quality dropped tomorrow. If you have fewer than three of those, you do not know if it is working. You have an opinion about it.

That is not a moral failing. It is the default state of almost every team that shipped an LLM feature in the last two years. The demo was convincing, the launch went fine, the logs are green. The question "is it working?" only becomes urgent the first time a customer pastes a screenshot of your agent confidently inventing something.

Why can't I just look at the logs?

Because your logs answer a different question. Application logs tell you whether requests completed. An agent can complete every request and still be failing: it can return HTTP 200 while summarizing an empty search result, call the wrong tool with perfect syntax, or quietly retry itself eight times before giving a polite non-answer. I wrote a whole field guide to these quiet failures in Your AI Agent Has the Same Red Flags as Your Ex, and the uncomfortable summary is that none of them look like errors.

Traditional monitoring was built for deterministic software: same input, same output, errors throw. Agents break all three assumptions. So the tooling that watches them has to look at content, not just status codes.

The five checks

1. Can you see a full trace of one real run? Pick a production request from yesterday. Can you see the prompt, every tool call with its arguments, every retrieval, what came back, and the final answer, in order, with timings? If reconstructing one run takes an engineer an afternoon of grepping, you fail this check.

2. Does anything judge the answers? A trace shows what happened; it does not say whether it was any good. That takes evaluations: automated checks that score outputs, the most useful being groundedness, whether the answer is supported by what the tools actually returned. If nothing scores your outputs, quality problems surface through customers.

3. Do you know what a run costs? Not the monthly bill, a run. Token spend attributed per request, per customer, per feature. Context bloat grows quietly: a prompt that started at 500 tokens ends the quarter at 24,000, and nobody notices until finance does.

4. Can you replay a bad run? When someone reports a wrong answer, can you load exactly what the agent saw and step through it? Without replay, every bug report starts an archaeology project.

5. Would you know if quality dropped tomorrow? A model upgrade, a prompt tweak, a data change. If your eval scores are not tracked over time with someone alerted on regression, your safety net is your users' patience.

What does a "yes" actually look like?

It is less exotic than the vendor landscape suggests. One tracing layer wired into the agent (usually one line of OpenTelemetry configuration), three to five evaluations that encode what "good" means for your use case, and a dashboard someone actually looks at. The long version of the argument, including why each layer needs the others, is in my explainer The Agent Observability Gap. If you want to try writing an evaluation prompt right now, my free tool Judgekit generates one from a pasted trace.

The point is not tooling maximalism. The point is that "is it working?" should be answerable by pointing at a screen, in plain language, by someone who is not the engineer who built it.

How do I get from zero to evidence?

Start with one trace. Instrument one production flow end to end and look at ten real runs. In my experience that first afternoon of looking produces the most uncomfortable and most valuable findings of the whole exercise: the retry loop nobody knew about, the tool that silently fails, the prompt that grew three thousand tokens of history.

Then add evals for the two or three failure modes you just found. Then put cost per run next to them. Sequence matters less than starting with real traffic instead of a test set that flatters the demo.

This is also, in the interest of transparency, exactly what I sell: the Agent Trust Sprint is two weeks where I do the instrumenting, the failure analysis on real traffic, and leave your team with the evals as a regression safety net, at a fixed price.

FAQ

How is this different from regular APM like Datadog? APM watches infrastructure: requests, latencies, error rates. It has no opinion about whether an answer was grounded in the retrieved documents or whether the agent picked a sensible tool. You need both layers; one does not replace the other.

Do I need this if my AI feature is just a chatbot, not an agent? The checks shrink but do not disappear. A single-prompt feature still hallucinates, still drifts after model updates, and still burns tokens. Traces get simpler; evals matter just as much.

Can my team build this themselves? Yes, and many should. The tracing layer is mostly configuration on open standards. The judgment layer is where teams stall, because writing evaluations that mean something requires deciding what "good" means for your product. That decision cannot be outsourced to a library.

What is the fastest honest signal I can get this week? Pull ten production runs and read them end to end. No tooling required beyond your existing logs. If reading ten runs is hard, that difficulty is itself the answer to "is it working?"