Concept guide · Agents, tools, and MCP
How do you evaluate an agent?
A slower explanation for when the short interview answer is not enough. Start with the intuition, then connect it to the technical idea and the way an interviewer expects you to reason about it.
ML area
Generative AI → Agent evaluation
Interview format
Advanced · Scenario
Question
GENAI-085
Start with the idea
In simple terms
Evaluate an agent on more than its final answer. Check whether it selected the right tools, used correct arguments, stopped at the right time, recovered from failures, respected permissions, and stayed within budget.
Make it concrete
Two agents may produce the same answer, but one needed twelve unnecessary searches and exposed a restricted tool in its trace. Task success alone would miss that difference.
Build the answer
How to approach it
- 1Define the task outcome and the safe paths that can achieve it.
- 2Score actions, arguments, recovery, budgets, and final results separately.
- 3Keep production failures as regression cases.
Go one level deeper
The technical explanation
Build trace-level evaluations with expected tool calls or acceptable alternatives, state-transition checks, failure injections, and human-labelled outcomes. Track success, retries, loops, latency, cost, safety violations, and escalation quality. If an LLM judge is used, calibrate it against human examples and retain representative traces for review.
Say this in the interview
Measure task success, tool selection, argument correctness, path efficiency, recovery, safety, cost, latency, and human handoffs. Read traces, classify failure modes, test tool and environment failures, and validate any automated judge against human-labelled tasks.
Keep preparing