Grading an Agent's Tool Use: A Worked Evaluation Example

Most evaluation writeups focus on whether an agent's final answer was correct. That's necessary, but it misses a second, quieter failure mode: an agent can land on a correct, well-personalized answer while still violating the operating instructions it was given for getting there. Below is a worked example, based on a real grading pass on a single agent turn, that shows what that looks like in practice.

The scenario

The agent was Perplexity running on Gemini 3.1 Pro, answering a location-recommendation question for a user with known equipment preferences. It had five tools available: search_web, fetch_url, execute_code, load_skill, and search_user_memories.

What it did well

• search_user_memories: Correctly used to recall that the user already owns their board and prefers wind-protected, flat-water spots over kitesurfing areas, rather than asking the user to repeat context they'd already given.
• search_verticals: Used correctly to find candidate locations near the target area.
• Tone and structure: Clear, active language; markdown headers stayed under a six-word limit; citations were placed inline immediately after the sentences they supported (with one exception, below).
• Follow-up question: The clarifying question focused on a single axis (wave preference vs. flat water), which is the right scope for a follow-up: one variable, not three.

Where it broke its own rules

Tool misuse. The instructions were explicit: don't use execute_code for "simple arithmetic, basic data display, printing raw data without processing, or tasks that can be accomplished with plain text responses." The agent used execute_code to write a hardcoded JSON blob of two locations to a CSV file. There was no computation, transformation, or analysis happening. A plain-text list or a markdown table would have carried the same information without invoking a code environment that exists for actual data processing.

Section structure. The formatting spec called for "2-3 well-cited sentences" between markdown headers before dropping into bullets. The agent skipped straight to bulleted lists under each header, losing the connective explanation the format was designed to provide.

Citation placement. One citation ([code_file:45]) was placed on its own line instead of inline after the sentence it supported. It's a small thing, but it's exactly the kind of formatting drift that compounds across a long conversation if it isn't caught.

Why this matters for building agents

None of these are reasoning failures. The agent found the right places and personalized correctly. They're instruction-following failures: using a tool outside its stated purpose, and drifting from a formatting spec under no real pressure to do so. If your evaluation rubric only scores "was the final answer good," you'll systematically miss this category of defect, and it's often the one users notice first (an agent that dumps hardcoded output into a CSV for no reason reads as either broken or padding its own tool-use metrics).

The fix isn't a smarter model. It's a grading pass that checks tool-use instructions and formatting constraints as separate, explicit rubric items, independent of whether the underlying answer was correct. Treat "did it use the right tool for the job" as its own pass/fail line, not a footnote under "the answer was fine."