Evals and trust for AI agents, in six moving pictures
How an agent works, what a trace records, and how an LLM judge grades it, animated scene by scene, then the five steps to build your first hallucination check.
Watch the explainer →Short, practical notes on making AI agents provably work in production, and on adopting AI without the chaos. The same thinking behind the fixed-price sprints.
How an agent works, what a trace records, and how an LLM judge grades it, animated scene by scene, then the five steps to build your first hallucination check.
Watch the explainer →A complete AI adoption plan for a small company: map usage, set the rules, pilot three use cases against baselines, and a scale-or-stop gate, in 90 days.
Read the note →The groundedness LLM metric scores whether every claim in an AI answer is supported by retrieved evidence. How it works, and what score to require.
Read the note →The best AI use cases for a small business: content drafting, routine admin, and first-response triage. Why these three first, and where not to start.
Read the note →AI agent failure modes that never throw errors: runaway loops, wrong tool calls, ungrounded answers, cost leaks, prompt injection. How to catch each one.
Read the note →AI readiness assessment cost runs $2,000-8,000 for small companies, more with consultant overhead. What the money buys and how to spend less for more.
Read the note →LLM evals are automated checks that score your AI's outputs against what "good" means for your product. What they are, the two main kinds, and your role.
Read the note →Shadow AI means employees using unapproved tools like personal ChatGPT accounts at work. Why bans fail, and the three moves that actually contain it.
Read the note →An AI agent audit examines architecture, prompts, and tool calls, runs failure analysis on real traffic, and leaves evals behind. Scope and fair pricing.
Read the note →An AI usage policy for a small business fits on one page: approved tools, banned data, account rules, an owner, and a review date. Here is what to write.
Read the note →A practical test for product leaders: five checks that show whether your AI agent works in production, with evidence instead of vibes.
Read the note →Occasional field notes on making AI agents provably work in production. No pitch, unsubscribe anytime.
Double opt-in via Buttondown.