foundry / anvil github.com/zaakirio/anvil

Anvil

Grounded answers, forged from cited sources.

Support agents fail quietly: they hallucinate answers, cite nothing, call the wrong tool, or post things nobody approved. Anvil is a support agent over the real FastAPI documentation, built evals first, so every one of those failure modes is a number CI can watch. Point the ingestion at any markdown docs tree, rebuild the goldens, and the same measurement system holds.

View on GitHub Quickstart
Anvil banner: four numbered citation nodes converge onto a bright strike point on an anvil silhouette, next to the wordmark and the tagline 'Grounded answers, forged from cited sources'.
0.5673
recall@5
0.7404
recall@10
0.5658
MRR
0.5847
nDCG@10
+10
recall@5 points added by the cross-encoder reranker

source: anvil eval retrieval · 52 real user questions (Stack Overflow + GitHub issues) over 1,199 chunks of the real FastAPI docs · hybrid search + cross-encoder rerank, fallback embedder all-MiniLM-L6-v2 (the keyless CI arm)

How it works

A LangGraph agent with a SQLite checkpointer, budget guards on every run, and a hard write boundary. Retrieval is hybrid: Postgres full-text and pgvector cosine each return 50 candidates, Reciprocal Rank Fusion merges them, and a cross-encoder reranks the top 25 on CPU. The top rerank score doubles as a grounding-confidence signal.

  1. 01
    Route

    claude-haiku-4-5 classifies the message: question, action, or smalltalk.

  2. 02
    Retrieve and rerank

    FTS + pgvector, RRF fusion, then ms-marco-MiniLM-L-6-v2 cross-encoder rerank. Heading-aware chunks give citation targets like tutorial-cors#use-corsmiddleware.

  3. 03
    Answer or refuse

    Above the confidence threshold, claude-sonnet-5 answers with citations required. Below it, a deterministic refusal fires without spending an LLM call.

  4. 04
    Act behind a human

    get_github_issue fetches live issues; draft_issue_comment is held behind a LangGraph HITL interrupt until a human approves. Nothing is ever posted to GitHub from this codebase.

  5. 05
    Gate

    anvil gate compares every eval run against a committed baseline and exits nonzero when any metric drops more than 0.02. CI runs it on every push.

anvil eval retrieval --embedder fallback
# hybrid + cross-encoder rerank, 52-query golden set
recall@5   0.5673
recall@10  0.7404
MRR        0.5658
nDCG@10    0.5847

# fusion-only (no reranker), same 52 queries
recall@5   0.4615
recall@10  0.6250
MRR        0.4077
nDCG@10    0.4385

# reranker is worth ~10 recall points at k=5
# and ~15 MRR points on this corpus

The eval stack is the product

Two layers plus a gate, and the numbers are deliberately unglamorous. The questions are real (vague titles, error messages, XY problems) and the corpus is real (1,199 chunks with heavy topical overlap), so the scores are what retrieval actually looks like before you tune it, measured honestly, with a gate that catches regressions from here.

  1. L1
    Retrieval, keyless, gates CI

    52 golden queries: 45 mined from Stack Overflow titles, 7 from FastAPI GitHub issues, each carrying its source URL. Relevant chunks labeled by inspection; every label is validated against the ingested corpus first, so a renamed heading fails loudly.

  2. L2
    End-to-end agent, fixture-proven

    42 golden conversations: 16 grounded answers with expected citations, 10 refusals on questions the docs genuinely cannot answer, 10 GitHub-issue lookups with expected arguments, 5 HITL comment drafts (approve and reject paths), 1 smalltalk. Deterministic checks first, then a claude-sonnet-5 judge scores faithfulness and relevancy.

  3. L3
    The gate

    GitHub Actions runs lint, the 50-test suite, ingestion, the keyless retrieval eval, and anvil gate on every push. A regression past tolerance goes red.

refusal calibration · measured on the real corpus
# rank-1 rerank score distributions, real data
answerable (52):  median 4.4, 5 of 52 below 0
known gaps (10):  range -9.7 to 4.6, 4 of 10 below 0

deterministic threshold: 0.0 (the cross-encoder's
own relevance boundary)

# the populations overlap; 3 of 10 gap questions
# score above threshold and rely on the
# prompt-layer refusal. two layers by design.

Every LLM call writes node, model, and exact token counts to a JSONL ledger, so anvil report prints cost per conversation from counted tokens, not a guess. A stdio MCP server (anvil-mcp) exposes search_docs and get_github_issue against the same backend; comment drafts stay agent-only because they belong behind the HITL interrupt. Generation is provider-agnostic through init_chat_model: Anthropic by default, or set ANVIL_ANSWER_MODEL to an openai:, google_genai:, or other provider-qualified model.

Quickstart

The committed corpus snapshot (118 pages of the FastAPI docs at a pinned commit) means no network and no API keys are needed for any of this.

terminal
$ docker compose up -d postgres
$ uv sync
$ uv run anvil ingest --embedder fallback   # keyless; --embedder openai with a key
$ uv run anvil eval retrieval --embedder fallback
$ uv run anvil gate
$ uv run pytest                              # 50 tests

# with ANTHROPIC_API_KEY set:
$ uv run anvil ask "How do I enable CORS?"
$ uv run anvil chat
$ uv run anvil eval agent
$ uv run anvil report

Honest limitations

  • The headline retrieval numbers are from the fallback embedder on a 1,199-chunk corpus. They establish the harness and the CI gate, not a claim about tuned production retrieval; the obvious headroom (stronger embedder, query rewriting, chunk-size tuning) is what the gate exists to measure.
  • The agent-eval layer is fully wired and fixture-proven, but the numbers it produces require API keys this environment does not have. RUN_WITH_KEYS.md is the one-command path once they exist.
  • The refusal threshold does not cleanly split answerable from unanswerable on real data: three of the ten golden gap questions score above it and depend on the prompt-layer refusal.
  • The LLM judge inherits judge variance; deterministic checks are separated precisely so the gate can hold them to a tighter tolerance than judge scores.
  • At this corpus size pgvector runs exact scans (no ANN index), which is the right call here and would need revisiting at 100x scale.
  • draft_issue_comment records and resolves drafts but never posts to GitHub; wiring the authenticated post-after-approval call is deliberately out of scope.
  • Golden labels are one person's judgment of relevance; the per-query output in eval_results/ exists so any label can be audited against its source URL.

This section is not small print. A retrieval number without its failure modes is an advert, and this page is a datasheet.