View a markdown version of this page

Skill evaluators - Amazon Bedrock AgentCore
Builtin.SkillSelectionAccuracyBuiltin.SkillInstructionFollowingFramework supportSkipping behaviorSkill placeholders for custom evaluators

Skill evaluators

A skill is a reusable SKILL.md instruction file that an agent loads at runtime to help with a task. Skills follow the open Agent Skills standard for skill file structure. For terminology, see Skill.

AgentCore Evaluations provides two built-in evaluators for agents that use skills. Both are tool-level evaluators. AgentCore Evaluations emits one result per skill invocation and anchors each result to the tool call span that loaded the skill. A session with three skill invocations produces three results per evaluator.

You can use these evaluators with on-demand, batch, and online evaluations. For the exact prompt bodies, see Skill selection accuracy and Skill instruction following on the Prompt templates page.

Builtin.SkillSelectionAccuracy

The Skill selection accuracy evaluator (Builtin.SkillSelectionAccuracy) judges whether the skill the agent loaded fits the task, given the catalog of available skills. It returns Yes (1.0) or No (0.0).

Placeholders the evaluator uses:

  • invoked_skill – The name of the skill the agent loaded in this tool call.

  • available_skills – The catalog of skills the agent could choose from at runtime. Not every framework exposes a catalog. When the catalog is not in the trace, this placeholder is empty and the evaluator judges the invoked skill against the user request and the conversation context alone.

  • user_message – The user request in the turn that triggered the skill invocation.

  • context – Previous turns up to the tool call being evaluated.

The evaluator runs whenever AgentCore Evaluations detects a skill invocation. It focuses on the selection decision, not on how well the agent then executed the skill.

Builtin.SkillInstructionFollowing

The Skill instruction following evaluator (Builtin.SkillInstructionFollowing) judges how fully the agent followed the loaded skill’s prescribed steps. It returns Fully Followed (1.0), Mostly Followed (0.75), Partially Followed (0.5), Minimally Followed (0.25), or Not Followed (0.0).

Placeholders the evaluator uses:

  • invoked_skill – The name of the skill the agent loaded in this tool call.

  • skill_content – The full body of the loaded skill’s SKILL.md instructions.

  • context – The full session context — every turn from session start through session end. Because the agent may carry out prescribed steps at any point after loading the skill, the judge needs the entire session, not just the turns up to the tool call.

The evaluator runs only when both the invoked skill and its SKILL.md body are present in the trace. The judge identifies the prescribed steps in the skill body, then, for each step, determines from the conversation record whether the step was fully carried out, partially carried out, or skipped, and produces an overall rating consistent with that per-step breakdown.

Framework support

AgentCore Evaluations detects skill invocations from two kinds of trace signals: a filesystem read of a SKILL.md file, and a framework’s native skill-loading tool. The filesystem-read signal works for any framework. The native-tool signal applies to the specific frameworks in the second row of the following table.

Detection method

Frameworks

SKILL.md file read (universal)

LlamaIndex, OpenAI Agents, and any agent that reads SKILL.md through a generic file-read tool.

Native skill-loading tool (in addition to file read)

Strands Agents, LangGraph Deep Agents, Google ADK, Claude Agent SDK.

For the file-read path, AgentCore Evaluations matches a tool call as a skill load when its parameters contain a path ending in /SKILL.md and the tool result body is a well-formed SKILL.md file (frontmatter with name and description fields, followed by an instructions body).

An available-skills catalog is emitted natively by Strands Agents, LangGraph Deep Agents, and Google ADK. Claude Agent SDK and file-read agents do not emit a catalog. Builtin.SkillSelectionAccuracy still runs on those agents, with an empty available_skills placeholder.

For general instrumentation setup and per-framework scope names, see Supported agent frameworks.

Skipping behavior

AgentCore Evaluations classifies a tool call as a skill invocation when the trace exposes invoked_skill or skill_content for that call. If neither signal is present, AgentCore Evaluations skips both skill evaluators for that call and emits no result. If your agent doesn’t load any skills, both evaluators produce zero results for the session — this is expected, not an error.

If your agent does load skills but the evaluators still return no results, run the diagnostic skill in Diagnostic skill source to confirm the traces carry the expected signals.

Skill placeholders for custom evaluators

You can write a custom TOOL_CALL evaluator that reasons about the skills an agent loaded. The skill placeholders described above (invoked_skill, skill_content, available_skills, user_message) are available to any custom TOOL_CALL evaluator alongside the standard tool-level placeholders (context, available_tools, tool_turn). Session-level and trace-level custom evaluators cannot use these placeholders.

How AgentCore Evaluations behaves depends on which skill placeholders the custom evaluator references:

  • An evaluator that references invoked_skill behaves like Builtin.SkillSelectionAccuracy: it runs only on skill-invocation tool calls, and {context} renders the standard pre-call snapshot.

  • An evaluator that references skill_content behaves like Builtin.SkillInstructionFollowing: it runs only on skill-invocation tool calls whose SKILL.md body is available, and {context} renders the full session context — every turn from session start through session end. This differs from the standard tool-level context.

In either case, AgentCore Evaluations emits one result per skill invocation. For custom-evaluator authoring, see Create evaluator.