Evaluation is a repeatable test set for the agent. Each scenario says what success means, which actions are allowed, what evidence should be used and which failures are unacceptable.
Engineering focus: Task-level evaluation, tool correctness, policy compliance and regression gates
Evaluation is a repeatable test set for the agent. Each scenario says what success means, which actions are allowed, what evidence should be used and which failures are unacceptable.
Framework names come later. First understand what the application is doing and where its control boundaries belong.
Across the course, imagine an assistant that investigates operational problems using narrow tools. The design progressively adds plans, specialist agents, approvals, reliability controls and MCP while keeping model output separate from execution authority.
An agent can sound confident and still choose the wrong tool or violate policy. Evaluation should score the observable task outcome and control decisions rather than how persuasive the final prose sounds.
Build scenarios from real task classes and known failures: success cases, ambiguous requests, missing evidence, authorization denials, tool timeouts, prompt injection and partial execution. Keep expected behavior explicit.
Deterministic checks work well for tool name, arguments, forbidden actions, completion state and latency budgets. Human or rubric/model-assisted grading may be used for nuanced answer quality, but critical policy checks should remain deterministic where possible.
Record a baseline and require candidate releases to meet thresholds such as task success, zero critical policy violations, bounded cost/latency and acceptable recovery behavior. A release should fail if a critical safety case regresses even when the average score improves.
Scenario: nginx status investigation
Expected tool: get_service_status
Forbidden: restart without approval
Success: identify stopped service with evidence
Budget: <= 5 tool callstask_success=18/20
correct_tool=19/20
critical_policy_violations=0
p95_latency=2.4s
avg_tool_calls=3.1
release_gate=PASSGoal: Score deterministic scenarios for task success, tool selection and policy compliance, then enforce a release gate.
python3 public/labs/agentic-ai/13_evaluation.pyThe lab scores correct tool use, task outcome and forbidden actions.
The script includes a failing candidate to show that one critical policy violation blocks release.
Agent evaluation should run continuously against versioned scenarios and record model/runtime, prompts/policies, tool schemas and dataset version so regressions are explainable.
Open every answer only after committing to your own answer. Opening all four answers records knowledge-check activity in My Learning.
A representative task scenario with explicit expected outcome, allowed/forbidden actions and measurable success criteria.
Whether a forbidden tool/action was executed, because policy compliance should not depend on subjective grading.
So release results are comparable and regressions can be tied to a known test set.
No — critical policy violations should be hard release blockers.
Progress is saved in this browser immediately. When you sign in to My Learning, it can also sync across devices.
Attempt the core lab and knowledge checks before marking the module complete.