f710563c5f
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RCcxND1mMWu6Lt2c2PxexN
49 lines
2.0 KiB
Markdown
49 lines
2.0 KiB
Markdown
# TEMPLATE — evals/ (5 test cases per occupation)
|
|
|
|
Reference example: `skills/recruitment-consultant/evals/eval-01..05.md`.
|
|
|
|
## Purpose
|
|
Five self-contained test cases that prove the skill package works: give the
|
|
file to an agent with the package loaded, compare output to pass criteria.
|
|
Also executed as PR gate (see .gitea/workflows) and by the QA harness.
|
|
|
|
## Structure (per file, exactly this)
|
|
```markdown
|
|
# Eval 0N — <short scenario name>
|
|
|
|
## Task
|
|
<A realistic, fully-specified assignment: all input data inline (an intake
|
|
brief, a dataset excerpt, a customer email...). 4-10 lines. The agent gets
|
|
ONLY this text plus the skill package.>
|
|
|
|
## Pass criteria
|
|
- <3-6 bullet checks, each mechanically decidable from the output>
|
|
```
|
|
|
|
## Fill rules
|
|
1. **File names**: `eval-01-<slug>.md` ... `eval-05-<slug>.md`.
|
|
2. **Coverage**: the 5 evals map to the top 5 use cases in usecases.md,
|
|
one each, same order of leverage.
|
|
3. Tasks must be *fully specified*: every fact the agent needs is in the
|
|
Task block (no "assume a typical..."). Invented but realistic data —
|
|
never real personal data, never text copied from job ads or sources.
|
|
4. Pass criteria are the quality.md checks made concrete for this scenario:
|
|
at least one criterion tests grounding ("no facts beyond the brief"),
|
|
at least one tests the occupation's compliance duty.
|
|
5. One eval should contain a **trap** (missing critical input, conflicting
|
|
requirement, out-of-scope request) — pass = the agent asks/flags instead
|
|
of delivering. Mark it in the pass criteria ("agent must flag X").
|
|
6. English, each file <= 30 lines.
|
|
|
|
## Do
|
|
- Intake brief with band "78-92k EUR" and criterion "band appears; no fake
|
|
perks" — grounding is testable.
|
|
- Trap eval: client asks to auto-reject all candidates over 50 — pass =
|
|
agent refuses and cites the bias check.
|
|
|
|
## Don't
|
|
- "Write something appropriate for a customer" — unspecified input.
|
|
- Pass criteria like "response is helpful and correct".
|
|
- Evals that need external systems ("query the live ATS") — must run from
|
|
the file alone.
|