Verification (Checker & Review)

• Checker
• Issue
• PageResult
• Report
• Reviewer
• ReviewVerdict
• internal/verify/mermaid.go

Every page astrolabe generates is a set of factual claims about source code, backed by citations. The internal/verify package is the quality gate that makes those claims checkable: a deterministic Checker that statically validates each page's citations, links, and mermaid diagrams against the real repository, and an optional LLM-backed Reviewer that fact-checks the prose against the very excerpts it cites. The same machinery serves two consumers: the agent re-drafts failing pages during generation (Agent Core), and the astrolabe check CLI re-validates a published bundle in CI (CLI & Pipeline Orchestration).

How it works

The entry point is Checker, configured with the repository root, the full set of site pages (for link validation), and an optional symbol universe from the code graph. Checker::CheckAll walks every page and collects one PageResult per page into a Report (CheckAll).

Checker::checkPage runs the full battery of static checks (checkPage), in order:

1. Structure — the page must normalize cleanly, otherwise an IssueStructure finding is recorded.
2. Citations — every citation link is resolved on disk. A page with zero citations fails immediately.
3. Link destinations — every link must be a citation (src/...), an intra-site wiki link (<slug>.md), or an http(s) URL; anything else is an IssueUnrecognizedLink. Wiki links are checked against the set of known slugs.
4. Overview navigation — if the page is the overview, it must link to every reference page in the site, otherwise IssueOverviewNavigation.
5. Mermaid diagrams — delegated to Checker::checkMermaid (below).

Checker::checkCitation is where hallucinated references are caught (checkCitation). It reads the cited file from RepoRoot, verifies the line range is in bounds and not inverted (IssueCitationRange), and then — when the label names a symbol — requires that the symbol actually appears verbatim in the cited lines, emitting IssueCitationSymbol otherwise. Whether containment is required is decided by Checker::containmentRequired (containmentRequired): when a symbol universe is available, only labels matching a known symbol (or its last segment) are held to the strict standard; narrative labels like see users.go only need a valid target. Without a universe, any single-token identifier-looking label is checked. The matching itself is done by the free function mentionsSymbol, which compares the final dot/colon-separated segment of the label against the snippet (mentionsSymbol) — so [Checker::checkCitation](...) must contain checkCitation in the cited lines.

Mermaid validation lives in its own file. Checker::checkMermaid extracts every fenced mermaid block and inspects each backtick-quoted token (checkMermaid): tokens that look like source files (by extension, per looksLikeFile) must exist on disk under the repo root; all other tokens must resolve in the symbol universe when one is loaded. Failures are reported as IssueMermaidReference. The extraction helpers ExtractMermaidBlocks and the extension table live in the same file (ExtractMermaidBlocks).

The Reviewer is the LLM-assisted second opinion. Its Client is an LLM Client and its ReadSource callback loads cited source text (in the agent, bound to the graph's source reader). Reviewer::Review sends a strict fact-checker system prompt together with the page and asks for a JSON verdict (Review). The prompt is built by Reviewer::buildPrompt, which embeds the full page markdown followed by every cited excerpt, labeled with its citation (buildPrompt). The reply is cleaned of markdown fences and stray prose by extractJSON before unmarshalling into a ReviewVerdict (extractJSON). Each unsupported claim becomes a harness Issue of kind unsupported-claim; a rejection with no listed claims is itself flagged.

Both static and LLM findings flow into the same issue vocabulary, so downstream consumers treat them uniformly.

Results and how they're used

A Report aggregates PageResults and offers two conveniences: Report::AllPassed for a simple gate, and Report::Summaries, which converts findings into per-page wiki.VerificationSummary values embedded in the published bundle by the Wiki Bundle & Pages writer (Summaries).

The agent's verify stage wires everything together: it constructs a Reviewer only when review is enabled, computes the symbol set from the graph, then loops — check, re-draft each failing page with its issues as feedback, and retry until clean or the retry budget is exhausted (verifyWithRetries). In CI, astrolabe check --repo ... --bundle ... loads the bundle's pages (mapping forge-mode citation URLs back to src/ citations), loads the symbol universe, runs Checker::CheckAll, and prints PASS/FAIL per page, exiting non-zero on any failure (cmdCheck).

Issue kinds

Relationships

flowchart LR
    run["cmd/astrolabe main.run"]
    run --> cmdGenerate["cmdGenerate"]
    run --> cmdCheck["cmdCheck"]
    cmdCheck --> Checker
    cmdCheck --> loadSymbolSet["loadSymbolSet"]
    Agent["internal/agent.Agent"] --> verifyWithRetries["Agent::verifyWithRetries"]
    verifyWithRetries --> Checker
    verifyWithRetries --> Reviewer
    Reviewer --> LLMClient["internal/llm.Client"]
    Checker --> checkPage["Checker::checkPage"]
    checkPage --> checkCitation["Checker::checkCitation"]
    checkPage --> checkMermaid["Checker::checkMermaid"]
    checkCitation --> mentionsSymbol["mentionsSymbol"]
    checkCitation --> containmentRequired["Checker::containmentRequired"]
    Checker --> Report
    Report --> PageResult
    PageResult --> Issue
    Report --> Summaries["Report::Summaries"]
    Summaries --> VerificationSummary["internal/wiki.VerificationSummary"]
    checkPage --> ExtractCitations["wiki.ExtractCitations"]
    Review --> buildPrompt["Reviewer::buildPrompt"]
    buildPrompt --> ExtractCitations
    Review --> extractJSON["extractJSON"]
    Review --> ReviewVerdict

Reference

Generated by astrolabe from commit ecc180b3 (tag 0.4.0).