The implementation story — a custom Python agent framework, Claude API orchestration, a 985-test suite, and the engineering decisions that keep the system reliable at scale.
| Category | Detail |
|---|---|
| Language | Python 3.12 |
| AI | Claude API (claude-sonnet-4-6 primary) |
| Orchestration | Claude Code + custom agent framework |
| Storage | Markdown + YAML frontmatter (primary), SQLite (telemetry), LanceDB (embeddings) |
| Content store | Obsidian vault |
| Testing | pytest — 985 tests |
| Scripting | 30+ custom Python scripts (ingest, generate, repair, lint) |
| Hosting | GitHub Pages (this site) |
The controller pattern: a main Claude Code session dispatches specialized subagents in parallel — each running in an isolated git worktree to prevent merge conflicts. Agents are ephemeral; the vault is the durable state. The controller reviews each agent's output before cherry-picking commits to main.
The /orchestrate ingestion command is a LangGraph StateGraph
of 8 nodes with one conditional entry gate (inbox_count > 0,
else short-circuit to END). All nodes share a PipelineState TypedDict whose
list fields use Annotated[list, add] reducers so each node
accumulates results into shared state without overwriting previous nodes' work.
The graph is wrapped in SqliteSaver so an interrupted run resumes from the
last completed node without redoing any work.
| Node | Role |
|---|---|
pre_flight |
Counts inbox files; branches to ingest if inbox_count > 0, else short-circuits to END — the conditional gate |
ingest |
Reads raw URL / clipping .md files from the inbox and creates structured save stubs |
heuristic_enrich |
Applies rule-based heuristics to assign resource_type, platform, and domain hints — no LLM call required |
llm_enrich |
Sends each stub to Claude Sonnet concurrently for title, summary, topics, and YAML frontmatter generation |
route_career |
Routes career-related saves to the personal-vault inbox rather than the research vault |
embed |
Generates vector embeddings for every save and upserts them into LanceDB for semantic retrieval |
topic_index |
Regenerates TOPIC-INDEX mini-indexes so BFS navigation is current on the next query |
finalize |
Writes the CHANGELOG entry (cost, save count, error count, run ID) and records per-run telemetry to SQLite |
LangGraph replaces an earlier ad-hoc loop with an explicit directed graph of typed nodes
and edges — making the pipeline structure readable and auditable at a glance. The shared
PipelineState TypedDict uses Annotated[list, add] reducers so
each node accumulates into shared state without overwriting previous nodes' work. The real
differentiator is SqliteSaver checkpointing: every completed node is persisted
to SQLite, so interrupted runs resume from the last checkpoint instead of starting over. A
recent production run hung mid-pipeline on a network read; after the timeout resolved, the
graph resumed cleanly from the last completed node, skipping already-processed saves with
no data loss.
Every new feature follows TDD: failing test first, then implementation, then green, then commit. The quality-RSI loop closes the feedback cycle automatically.
CHANGELOG / GAPS / SUGGESTIONS / INSIGHTS / NEXT-STEPS each live as a
folder of timestamped .md entries with a derived index.
Never overwritten; always queryable by future sessions.
L0 topic scan → L1 YAML frontmatter filter → L2 full read. Scales to 6K+ saves without reading everything; the filter eliminates 80–90% of reads before they happen.
Each concurrent agent works in a fresh git worktree. Cherry-pick to main when done — no merge conflicts, no clobbered in-progress work.
LanceDB vector embeddings for semantic retrieval; SQLite for per-run telemetry (cost, quality score, save count). Complement the markdown-first storage model.