docs: HANDOVER for Sonnet full-catalog run + decisions log
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RCcxND1mMWu6Lt2c2PxexN
This commit is contained in:
skillfactor-pipeline
85
HANDOVER.md
Normal file
85
HANDOVER.md
Normal file
@@ -0,0 +1,85 @@
|
|||||||
|
# HANDOVER — Fable 5 → Sonnet 4.6 (2026-07-07)
|
||||||
|
|
||||||
|
Quality foundation is done. Your job is the multi-day full-catalog run.
|
||||||
|
**Read `Sites(staging)\skillfactor\auftrag_sonnet.md`, follow
|
||||||
|
`pipeline/templates/` and `pipeline/prompts/` strictly — do not redesign
|
||||||
|
anything.** Design contracts: `docs/batch-architecture.md`.
|
||||||
|
|
||||||
|
## What Fable built today (all committed)
|
||||||
|
|
||||||
|
| Artefact | Where | State |
|
||||||
|
|---|---|---|
|
||||||
|
| Baseline inventory | `docs/inventory.md` | done |
|
||||||
|
| Extraction schema | `pipeline/prompts/extract_schema.json` | final |
|
||||||
|
| Extraction prompt (Ollama) | `pipeline/prompts/extract_posting.txt` | final, tested |
|
||||||
|
| Local extractor + validator | `pipeline/extract_local.py` | **done & tested: 20/20 clean, ø 6.5 s/ad on gemma3:27b** |
|
||||||
|
| Depth templates + style | `pipeline/templates/*.template.md`, `STYLE.md` | final |
|
||||||
|
| Acceptance bar | `pipeline/templates/QUALITY_BAR.md` | final |
|
||||||
|
| Gold standard | `skills/recruitment-consultant/` | reviewed & upgraded (Output lines, verified market %, TRAP eval) |
|
||||||
|
| Budget/resume module | `pipeline/progress.py` | done & tested (hard cap 33 000 JSearch req) |
|
||||||
|
| QA harness | `pipeline/qa_sample.py` | done & tested |
|
||||||
|
| Batch runner | `pipeline/batch_run.py` | **skeleton — you implement the TODOs** |
|
||||||
|
| Architecture design | `docs/batch-architecture.md` | final |
|
||||||
|
| `.env` | `OLLAMA_URL` / `OLLAMA_MODEL` added | works |
|
||||||
|
|
||||||
|
## Your exact start sequence
|
||||||
|
|
||||||
|
1. Read `auftrag_sonnet.md`, `docs/batch-architecture.md`,
|
||||||
|
`pipeline/batch_run.py` (the TODOs name their own acceptance criteria).
|
||||||
|
2. Apply the MSSQL change (batch-architecture.md §MSSQL) —
|
||||||
|
`occupation_slug` column + backfill + index.
|
||||||
|
3. Implement `stage_packages` (reuse `p2_generate.py`/`p4_publish.py`,
|
||||||
|
throttled ≥ 1 s/repo, check-before-create) and run Phase 1 for all
|
||||||
|
3 039 ESCO occupations.
|
||||||
|
4. Implement `stage_evidence` (generalize p3a per occupation, extraction
|
||||||
|
via `extract_local.py` as-is, incremental p3b, per-slug p3c).
|
||||||
|
Every JSearch HTTP call goes through `progress.spend_request()` FIRST.
|
||||||
|
5. After each occupation batch: `qa_sample.py --rate 0.02`; two consecutive
|
||||||
|
exit-1 = stop and diagnose. Depth files must pass QUALITY_BAR mechanical
|
||||||
|
checks before pushing.
|
||||||
|
6. Depth tier 1 (top-200) per templates; tier 3 as resumable night batch.
|
||||||
|
|
||||||
|
## Hard rules (from the finalize order — not negotiable)
|
||||||
|
|
||||||
|
- JSearch lifetime budget 33 000 requests — enforced in code, do not bypass.
|
||||||
|
- ≤ 2 req/s; cache first; thin market → synonyms → Adzuna → mark failed.
|
||||||
|
- Mass extraction (~180 k ads ≈ 14 GPU-days) runs as a **detached script**
|
||||||
|
(`run_in_background` / scheduled task), never inside your session loop.
|
||||||
|
- No secrets in code/logs. `.env` stays uncommitted.
|
||||||
|
- Commits small and conventional.
|
||||||
|
|
||||||
|
## Open points / caveats
|
||||||
|
|
||||||
|
1. **Git history was squashed today 11:21** into one commit (orphan branch
|
||||||
|
`clean-history` → renamed master) — not by Claude. Old history is kept
|
||||||
|
as tag `backup/pre-squash-history`. Ask Robert before deleting the tag;
|
||||||
|
if the squash was unintended, the old commits are all still there.
|
||||||
|
2. `ADZUNA_APP_ID/KEY` in `.env` are empty — the finalize order lists
|
||||||
|
values (§2); copy them in before the Adzuna fallback is needed.
|
||||||
|
3. Enrichment templates (curriculum/learning/literature/glossary/
|
||||||
|
capabilities) do NOT exist yet — Fable's day covered the four depth
|
||||||
|
files per auftrag_fable §3. Derive the enrichment ones from
|
||||||
|
STYLE.md + the recruiter examples when Phase 3 starts, same pattern.
|
||||||
|
4. `p2_generate.py` currently emits a fixed balanced set of 100 — needs the
|
||||||
|
per-occupation refactor (stage_packages TODO).
|
||||||
|
5. SKILL.md trigger descriptions come from crosswalk O*NET tasks and are
|
||||||
|
sometimes off-profile (recruiter shows HR-manager tasks). Known
|
||||||
|
generator limitation — fix in stage_packages if cheap, else log it.
|
||||||
|
6. Ollama box (91.184.179.239:11434) is unencrypted/unauthenticated —
|
||||||
|
COLOGNE2 firewall must restrict port 11434 to this webserver's IP
|
||||||
|
(flagged to Robert earlier, unverified).
|
||||||
|
7. `data/progress.json` is committed in its empty initial state; it is
|
||||||
|
run-state, commit it whenever you checkpoint a batch.
|
||||||
|
|
||||||
|
## Verification snapshot (what "working" looked like today)
|
||||||
|
|
||||||
|
```
|
||||||
|
.venv/Scripts/python.exe pipeline/extract_local.py \
|
||||||
|
--ads data/raw/jobs/recruiter_ads.json \
|
||||||
|
--out data/evidence/testset_fable.jsonl --limit 20
|
||||||
|
# -> done: 20 ok, 0 failed (0% fail)
|
||||||
|
.venv/Scripts/python.exe pipeline/qa_sample.py \
|
||||||
|
--extractions data/evidence/testset_fable.jsonl \
|
||||||
|
--ads data/raw/jobs/recruiter_ads.json --rate 0.10
|
||||||
|
# -> 2 sampled, 0 mechanical issues
|
||||||
|
```
|
||||||
32
docs/decisions.md
Normal file
32
docs/decisions.md
Normal file
@@ -0,0 +1,32 @@
|
|||||||
|
# Decisions log
|
||||||
|
|
||||||
|
Short rationale for choices that are not obvious from the code
|
||||||
|
(rule from auftrag_fable.md §Regeln).
|
||||||
|
|
||||||
|
## 2026-07-07 (Fable 5 — quality foundation day)
|
||||||
|
|
||||||
|
1. **Ollama structured output + strict prompt, both.** The JSON schema is
|
||||||
|
passed as `format` to `/api/generate` (server-side shape enforcement)
|
||||||
|
AND the prompt demands strict JSON with 3 few-shots. Belt and braces:
|
||||||
|
the schema pins keys/types, the few-shots pin content quality.
|
||||||
|
Result: 20/20 schema-valid on the first model pass (gemma3:27b, ø 6.5 s).
|
||||||
|
2. **Validator repairs instead of second prompt iteration.** Testset showed
|
||||||
|
95 % content-clean; the remaining defects (skill phrases leaking into
|
||||||
|
qualifications, mojibake from broken ad encodings, acronym casing) are
|
||||||
|
deterministic and cheaper to fix in `validate()` than by prompt tuning
|
||||||
|
on a 27B model. After hardening: 20/20 clean.
|
||||||
|
3. **`evidence_job` gets an `occupation_slug` column** (nullable ALTER +
|
||||||
|
backfill) instead of per-occupation tables or a rebuild-per-run design —
|
||||||
|
p3b becomes incremental MERGE; existing 350 recruiter rows stay valid.
|
||||||
|
4. **Mass extraction runs as a detached script, not in the Claude loop**
|
||||||
|
(~180 k ads x 6.5 s ≈ 14 GPU-days): Claude only spot-checks 2 % samples
|
||||||
|
via `qa_sample.py`. Budget enforcement lives in code
|
||||||
|
(`progress.spend_request()` raises), not in instructions.
|
||||||
|
5. **Templates as `pipeline/templates/*.template.md` + shared STYLE.md.**
|
||||||
|
Skeleton headings are greppable on purpose — QUALITY_BAR's mechanical
|
||||||
|
checks and the PR gates rely on exact section markers.
|
||||||
|
6. **Git history squash discovered mid-session** (orphan branch
|
||||||
|
`clean-history` renamed to master, 11:21 local — not done by Claude).
|
||||||
|
Old 16-commit history preserved as tag `backup/pre-squash-history`;
|
||||||
|
new work committed on top of the squashed master. Delete the tag once
|
||||||
|
the squash is confirmed intentional and nothing is missing.
|
||||||
Reference in New Issue
Block a user