skillfactor-pipeline/HANDOVER.md

HANDOVER — Fable 5 → Sonnet 4.6 (2026-07-07)

Quality foundation is done. Your job is the multi-day full-catalog run. Read Sites(staging)\skillfactor\auftrag_sonnet.md, follow pipeline/templates/ and pipeline/prompts/ strictly — do not redesign anything. Design contracts: docs/batch-architecture.md.

What Fable built today (all committed)

Artefact	Where	State
Baseline inventory	docs/inventory.md	done
Extraction schema	pipeline/prompts/extract_schema.json	final
Extraction prompt (Ollama)	pipeline/prompts/extract_posting.txt	final, tested
Local extractor + validator	pipeline/extract_local.py	done & tested: 20/20 clean, ø 6.5 s/ad on gemma3:27b
Depth templates + style	pipeline/templates/*.template.md, STYLE.md	final
Acceptance bar	pipeline/templates/QUALITY_BAR.md	final
Gold standard	skills/recruitment-consultant/	reviewed & upgraded (Output lines, verified market %, TRAP eval)
Budget/resume module	pipeline/progress.py	done & tested (hard cap 33 000 JSearch req)
QA harness	pipeline/qa_sample.py	done & tested
Batch runner	pipeline/batch_run.py	skeleton — you implement the TODOs
Architecture design	docs/batch-architecture.md	final
.env	OLLAMA_URL / OLLAMA_MODEL added	works

Your exact start sequence

1. Read auftrag_sonnet.md, docs/batch-architecture.md, pipeline/batch_run.py (the TODOs name their own acceptance criteria).
2. Apply the MSSQL change (batch-architecture.md §MSSQL) — occupation_slug column + backfill + index.
3. Implement stage_packages (reuse p2_generate.py/p4_publish.py, throttled ≥ 1 s/repo, check-before-create) and run Phase 1 for all 3 039 ESCO occupations.
4. Implement stage_evidence (generalize p3a per occupation, extraction via extract_local.py as-is, incremental p3b, per-slug p3c). Every JSearch HTTP call goes through progress.spend_request() FIRST.
5. After each occupation batch: qa_sample.py --rate 0.02; two consecutive exit-1 = stop and diagnose. Depth files must pass QUALITY_BAR mechanical checks before pushing.
6. Depth tier 1 (top-200) per templates; tier 3 as resumable night batch.

Hard rules (from the finalize order — not negotiable)

• JSearch lifetime budget 33 000 requests — enforced in code, do not bypass.
• ≤ 2 req/s; cache first; thin market → synonyms → Adzuna → mark failed.
• Mass extraction (~180 k ads ≈ 14 GPU-days) runs as a detached script (run_in_background / scheduled task), never inside your session loop.
• No secrets in code/logs. .env stays uncommitted.
• Commits small and conventional.

Open points / caveats

1. Git history was squashed today 11:21 into one commit (orphan branch clean-history → renamed master) — not by Claude. Old history is kept as tag backup/pre-squash-history. Ask Robert before deleting the tag; if the squash was unintended, the old commits are all still there.
2. ADZUNA_APP_ID/KEY in .env are empty — the finalize order lists values (§2); copy them in before the Adzuna fallback is needed.
3. Enrichment templates (curriculum/learning/literature/glossary/ capabilities) do NOT exist yet — Fable's day covered the four depth files per auftrag_fable §3. Derive the enrichment ones from STYLE.md + the recruiter examples when Phase 3 starts, same pattern.
4. p2_generate.py currently emits a fixed balanced set of 100 — needs the per-occupation refactor (stage_packages TODO).
5. SKILL.md trigger descriptions come from crosswalk O*NET tasks and are sometimes off-profile (recruiter shows HR-manager tasks). Known generator limitation — fix in stage_packages if cheap, else log it.
6. Ollama box (91.184.179.239:11434) is unencrypted/unauthenticated — COLOGNE2 firewall must restrict port 11434 to this webserver's IP (flagged to Robert earlier, unverified).
7. data/progress.json is committed in its empty initial state; it is run-state, commit it whenever you checkpoint a batch.

Verification snapshot (what "working" looked like today)

.venv/Scripts/python.exe pipeline/extract_local.py \
    --ads data/raw/jobs/recruiter_ads.json \
    --out data/evidence/testset_fable.jsonl --limit 20
# -> done: 20 ok, 0 failed (0% fail)
.venv/Scripts/python.exe pipeline/qa_sample.py \
    --extractions data/evidence/testset_fable.jsonl \
    --ads data/raw/jobs/recruiter_ads.json --rate 0.10
# -> 2 sampled, 0 mechanical issues