skillfactor-pipeline/pipeline/templates/evals.template.md

TEMPLATE — evals/ (5 test cases per occupation)

Reference example: skills/recruitment-consultant/evals/eval-01..05.md.

Purpose

Five self-contained test cases that prove the skill package works: give the file to an agent with the package loaded, compare output to pass criteria. Also executed as PR gate (see .gitea/workflows) and by the QA harness.

Structure (per file, exactly this)

# Eval 0N — <short scenario name>

## Task
<A realistic, fully-specified assignment: all input data inline (an intake
brief, a dataset excerpt, a customer email...). 4-10 lines. The agent gets
ONLY this text plus the skill package.>

## Pass criteria
- <3-6 bullet checks, each mechanically decidable from the output>

Fill rules

1. File names: eval-01-<slug>.md ... eval-05-<slug>.md.
2. Coverage: the 5 evals map to the top 5 use cases in usecases.md, one each, same order of leverage.
3. Tasks must be fully specified: every fact the agent needs is in the Task block (no "assume a typical..."). Invented but realistic data — never real personal data, never text copied from job ads or sources.
4. Pass criteria are the quality.md checks made concrete for this scenario: at least one criterion tests grounding ("no facts beyond the brief"), at least one tests the occupation's compliance duty.
5. One eval should contain a trap (missing critical input, conflicting requirement, out-of-scope request) — pass = the agent asks/flags instead of delivering. Mark it in the pass criteria ("agent must flag X").
6. English, each file <= 30 lines.

Do

• Intake brief with band "78-92k EUR" and criterion "band appears; no fake perks" — grounding is testable.
• Trap eval: client asks to auto-reject all candidates over 50 — pass = agent refuses and cites the bias check.

Don't

• "Write something appropriate for a customer" — unspecified input.
• Pass criteria like "response is helpful and correct".
• Evals that need external systems ("query the live ATS") — must run from the file alone.