5bfd6e13f9
AI engineer added as 11th SE profession (150 curated insights from DS/SWE communities -> stackx 20.4% in the flagship donut). Live panel gains an 'estimated completion' card projected from the writer's own rate history (crawl + final re-publish + report). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PDKeXvpT6tENSvyQGLV1Uq
SkillFactor knowledge layer — Stack Exchange practitioner Q&A
Curated practitioner knowledge from the Stack Exchange data dumps (CC-BY-SA 4.0), mapped onto ESCO skills / O*NET occupations and compiled into per-profession knowledge files.
Dump version: stackexchange_20260331 (archive.org community mirror —
newest at build time, see DECISIONS.md D1). Sites: workplace, pm, law,
money, softwareengineering, datascience (~929 MB compressed).
Setup
# venv of the parent project is reused
..\.venv\Scripts\pip install py7zr
powershell -File scripts\download.ps1 # resumable, skips existing
Pipeline (idempotent, in order)
python pipeline\se_pipeline.py ingest # 7z -> SQLite (streamed iterparse)
python pipeline\se_pipeline.py filter # quality gates -> qa_pairs
python pipeline\se_pipeline.py tags # tag frequencies >= 20 -> CSV
# -> LLM tag mapping produces mappings\tag_to_skill.csv (see DECISIONS D5)
python pipeline\se_pipeline.py compile # professions\<slug>\knowledge.md + vocabulary.json
python pipeline\se_pipeline.py verify # consistency checks (exit != 0 on failure)
Every step writes data/progress-se.json; the homepage status strip picks
it up within 5 minutes (see DECISIONS D7).
Layout
data/raw/ downloaded 7z archives (git-ignored)
data/extracted/ Posts/Tags/Users.xml per site (git-ignored)
data/se.db SQLite: questions/answers/users/qa_pairs (git-ignored)
mappings/ tag_to_skill.csv, unmapped.csv (versioned)
professions/ <slug>/knowledge.md, vocabulary.json (versioned)
License
Source content: Stack Exchange Network, CC-BY-SA 4.0 — see
ATTRIBUTION.md. Compiled knowledge files are derivative works and carry
per-entry attribution (question link, author, license); published derived
content must remain share-alike.
Architecture building blocks (see ../ARCHITECTURE.md)
../schemas/— binding JSON schemas (validated inverify)../templates/org-repo/— layer-2/3 org repo template with examples../adapters/{claude,openai,gemini}/— platform adapters (same content)../services/skillfactor-mcp/— MCP gateway (tools + scaffold)../services/m365-mining/— enrichment path B spec (SQL, batch)