AlienMachine CMS

Deep-Research Agent Architecture: Graph Runtime, Checkpointing, Governance, and Benchmarks

A production-grade deep-research agent can outperform human research teams on verified long-form reports when it is engineered around machine-native strengths rather than modeled as a single conversational analyst. The decisive shift is from "chatbot plus search" to a checkpointed graph-of-agents system: many specialized agents execute bounded research, source-integrity checks, counter-evidence hunts, citation validation, synthesis, and review as explicit graph nodes with durable state, deterministic replay, tool-call provenance, and human governance gates. The advantage comes from doing what humans cannot reliably do at scale — running hundreds of search/fetch/read/verify/re-plan loops without fatigue, preserving every intermediate artifact, forcing every sentence-level assertion through provenance checks, and rerunning the same pipeline deterministically after model, prompt, cache, or source changes.

The strongest design is not a fully autonomous free-for-all. It is a governed pipeline that exploits large-scale parallelism while constraining failure modes through typed artifacts, evidence contracts, adversarial source screening, claim-level verification, independent reviewer passes, and replayable decision logs. A strong reasoning model acts as the top-level planner, synthesizer, and policy adjudicator, while cheaper or narrower workers handle fan-out search, document triage, extraction, contradiction finding, citation checks, and formatting. The runtime should be a graph with checkpointed state, bounded loops, budget-aware scheduling, and structured outputs at every edge.

The report design should be claim-first: every claim in the final report traces to evidence records; every evidence record preserves source identity, retrieval metadata, document hashes or stable snapshots, quoted support spans, extraction confidence, and adversarial risk flags. Factuality is evaluated not only by whether cited sources support statements but by whether the system searched for and incorporated credible counter-evidence. The main risks — source poisoning, SEO spam, fabricated citations, citation laundering, stale sources, source impersonation, overconfident synthesis, cost explosions, and governance gaps — are manageable when provenance, counter-evidence, and replay are central primitives rather than late-stage QA checks.

This article is the architecture pillar: it owns the system layer — the checkpointed graph runtime, the typed source/evidence/claim data model, role fan-out and fan-in, deterministic replay, adversarial source screening, human governance gates, and evaluation benchmarks. The retrieval and verification algorithms the nodes run — BM25 and dense/hybrid retrieval, reciprocal rank fusion and MMR diversity selection, cross-encoder reranking, and NLI-based claim verification — are the subject of the companion pillar, Deep-research retrieval and claim-verification algorithms. Where this article touches those algorithms, it summarizes and links there for the math.

Design Objective

The objective is a verified research system that produces long-form reports with stronger factual grounding, broader source coverage, better counter-evidence recall, and more reproducible audit trails than a human team under ordinary time constraints. It should optimize five measurable capabilities:

  1. Verified synthesis — report claims are explicitly grounded in cited evidence and rejected when support is weak.
  2. Adversarial robustness — the system actively detects SEO spam, poisoned documents, fabricated references, source impersonation, and citation chains that do not reach primary evidence.
  3. Graph-scale research — planning, search, reading, extraction, contradiction, review, and synthesis are decomposed into parallel graph nodes with typed state.
  4. Deterministic auditability — every run can be replayed from checkpoints, with preserved inputs, outputs, tool calls, source metadata, and decisions.
  5. Production efficiency — model, search, fetch, cache, and review budgets are allocated by claim risk and report value rather than spent uniformly.

Core Architecture

The recommended architecture is a graph pipeline with explicit state transitions. It should not depend on an implicit conversation transcript as its source of truth; conversation can remain the operator interface, but durable state should be structured files or a checkpoint database. A high-level graph contains these stages:

  1. Intake and scope normalization
  2. Research plan generation
  3. Query strategy expansion
  4. Parallel search and discovery
  5. Source triage and integrity screening
  6. Document fetching and canonicalization
  7. Evidence extraction
  8. Claim candidate generation
  9. Counter-evidence and contradiction search
  10. Claim verification
  11. Synthesis planning
  12. Report drafting
  13. Citation audit
  14. Adversarial review
  15. Human-in-the-loop review
  16. Final render and provenance bundle

Each stage consumes and emits typed artifacts. The graph supports conditional branches, retries, reviewer-rejection paths, and budget gates — a high-risk claim automatically triggers deeper source discovery, primary-source preference, independent verifier review, and counter-evidence search, while a low-risk background claim requires fewer checks.

The orchestrator should separate roles:

Role Primary Responsibility Model/Tool Tier
Lead planner Research questions, decomposition, acceptance criteria, risk classes Strong reasoning model
Search workers Generate and execute diverse queries Cheap/medium model plus search tools
Source triage workers Rank sources, detect spam, identify authority and provenance Medium model plus rules
Reader/extractors Extract facts, quotes, tables, dates, caveats Medium model, sometimes OCR/PDF tools
Counter-evidence agents Search for disconfirming sources and minority views Medium/strong model
Claim verifier Map claims to evidence; reject unsupported assertions Strong reasoning model
Citation auditor Check source existence, citation-target match, quote accuracy, format Deterministic tools plus model
Adversarial reviewer Attempt to break the report via poisoning, impersonation, unsupported inference Strong reasoning model
Human reviewer Approve scope, risk policy, disputed claims, and publication Human gate

Agents should not merely "give opinions." They produce artifacts: query logs, source cards, evidence cards, contradiction cards, claim cards, verification decisions, budget decisions, and review findings.

Native Graph Orchestration

Graph orchestration is the backbone, because deep research is not a linear chain — it is an iterative search process with branching hypotheses, source conflicts, uncertainty, and re-planning. A proper graph runtime should support:

Capability Why It Matters
Durable checkpoints Resume long reports after crashes, rate limits, model errors, or human pauses
Node-level retries Rerun failed fetches or ambiguous extractions without restarting
Conditional routing Send high-risk or contradicted claims to deeper verification
Fan-out/fan-in Enable hundreds of parallel searches and readings followed by structured synthesis
Typed state Prevent ungrounded prose from becoming hidden state
Replay Enable regression testing and audit after prompt or model changes
Cost accounting Allow budget-aware scheduling by node, source, claim, and section
Human gates Support review before expensive expansion, disputed claims, or publication

The graph should encode reasoning as recoverable state, not just model hidden thoughts: the planner writes a research-plan artifact (questions, source preferences, exclusion criteria, risk classes, verification thresholds), and workers read it rather than inferring it from conversational context. A strong design allows heterogeneous execution — some nodes use model calls; others use deterministic code, search APIs, PDF parsers, cache lookups, citation parsers, embedding search, or validators. Model calls are one tool category inside a broader programmatic workflow.

Machine Advantages to Exploit

Design around asymmetries where machines exceed human teams:

  • Massive parallel fan-out. Agents can run hundreds of independent search strategies across phrasings, languages, source classes, time windows, and adversarial angles. The planner should intentionally generate query bundles for primary sources, academic sources, standards bodies, regulators, vendor docs, critical commentary, litigation, incident reports, and counter-evidence.
  • Tireless iteration. Agents loop until coverage thresholds are met, each loop asking: what remains unsupported? which claims are contradicted? which sources are low-quality? which sections lack primary evidence? what would change the conclusion?
  • Perfect state and recall. The system retains every source, quote, hash, extraction, rejection reason, and reviewer decision — enabling claim-level audit trails human teams rarely maintain.
  • Deterministic checkpointed replay. A run can answer which query found a source, which document version was read, which extract supported a claim, and what changed between runs.
  • Large-context synthesis. A top-level model can synthesize across a much larger evidence base than human working memory allows, provided evidence is structured, deduplicated, and ranked.
  • Programmatic tool composition. Search, fetch, OCR, PDF parsing, browser automation, citation parsing, checksum generation, archival, caching, vector retrieval, static validation, and rendering compose into one verifiable pipeline.

These advantages only pay off if the system resists collapsing everything into final prose too early. Delay prose generation until after source discovery, evidence extraction, counter-evidence search, and claim verification have produced structured inputs.

Research Planning

The first substantive stage converts the request into a research design explicit enough for independent agents to execute without hidden context. The plan should contain a research objective, audience and use, key questions, required sections, source hierarchy, claim risk classes, exclusion criteria, counter-evidence strategy, freshness requirements, budget policy, and human-review gates. The planner should distinguish stable claims (architectural principles) from time-sensitive claims (vendor capabilities, model names, pricing, leaderboards, tool APIs) that require current-source verification in live runs. It should also produce a coverage matrix of sections by evidence requirements, with minimum source diversity, preferred primary sources, expected counterarguments, and verification depth.

Search and Discovery Strategy

The search subsystem should perform systematic discovery through multiple strategies rather than one generic query: primary-source, academic and benchmark, standards and regulator, vendor documentation, incident and failure-mode, critical commentary, historical prior-art, counter-evidence, citation-chain backward and forward, domain-restricted, multilingual where relevant, time-windowed, exact-title and exact-quote, and source-identity verification searches.

Each query is recorded with parameters, timestamp, provider, returned URLs, ranking, snippets, and selection decisions — because search results are volatile and personalized, and without query logs the report cannot be audited. The system should generate query families (each with positive, negative, and adversarial formulations) rather than individual queries, and fan-out should be budget-aware: stop expanding a family when marginal sources become redundant or off-topic; expand when claims remain unsupported, sources disagree, or high-risk claims depend on weak evidence.

This section is the orchestration view of search — query logging, families, and budget. The retrieval and fusion algorithms each node runs (BM25, dense and hybrid retrieval, reciprocal rank fusion, diversity-aware MMR selection, and cross-encoder reranking) are covered in the companion retrieval and claim-verification algorithms article.

Source Triage

Triage happens before deep reading, to avoid spending model tokens on spam, copied content, fabricated sources, or low-authority commentary (unless studied as adversarial examples). Each discovered source receives a source card:

Field Description
URL and canonical URL Retrieval target and normalized identity
Publisher identity Organization, author, domain, ownership if knowable
Source type Primary, academic, standard, regulator, vendor, news, blog, forum, marketing, unknown
Publication date Freshness and chronology
Last modified/accessed Audit
Retrieval method Search result, direct fetch, crawl, archive, API, local file
Content hash Detects later changes
Authority score Source type, expertise, proximity, independence
Integrity risk score Spam signals, impersonation, generated content, suspicious redirects
Relevance score Topic and claim coverage
Read decision Read, skim, reject, archive only, adversarial sample
Rejection reason Required when excluded

Ranking should prefer primary evidence — official documentation, standards, peer-reviewed papers, regulator publications, benchmark definitions, public datasets, reproducible code, original incident reports. Secondary sources can provide interpretation but should not be sole support for technical claims when primary sources exist. Crucially, search rank is a discovery signal, not an authority signal: SEO-optimized pages often rank highly while being derivative or promotional.

Document Fetching and Canonicalization

Fetching is a first-class engineering problem. The fetch layer should capture HTTP metadata (status, headers, redirects, content type, timestamps), resolve canonical URLs, store snapshots, hash content, parse PDF and HTML, render JavaScript when needed, fall back to archives, detect language, remove boilerplate, extract tables, OCR scanned documents, and respect robots/terms policy. Canonicalization should avoid erasing evidence: preserve raw source artifacts and create cleaned views for model reading. The final report cites stable canonical sources, but the audit bundle retains retrieval details.

Evidence Extraction

Extraction produces structured evidence cards, not free-form notes:

Field Description
Evidence ID Stable identifier
Source ID Link to source card
Exact support span Quote or localized passage (within copyright-safe internal storage rules)
Paraphrased content Model-readable summary
Claim relevance Which question or claim it supports
Polarity Supports, contradicts, qualifies, contextualizes
Strength Direct, indirect, weak, background
Freshness Current, historical, stale, unknown
Extractor confidence Confidence in extraction, not truth
Caveats Scope limits, assumptions, definitions
Locator URL fragment, page, heading, table, or line
Verification status Pending, accepted, rejected, needs review

The extractor must distinguish "the source says X" from "X is true." That distinction prevents citation laundering, where a weak source's assertion becomes validated fact. Extraction should also preserve negative evidence — absence of support, qualifications, limitations, benchmark caveats, failures, and contradictions — which is essential for counter-evidence recall.

Claim-Centric Verification

The report is built from verified claim cards. A claim card represents one atomic assertion:

Field Description
Claim ID Stable identifier
Claim text Atomic assertion
Section target Where it may appear
Claim type Descriptive, causal, normative, comparative, quantitative, current, historical
Risk class Low, medium, high, critical
Evidence required Minimum support standard
Supporting evidence IDs Direct evidence
Contradicting evidence IDs Counter-evidence
Verification decision Pass, fail, uncertain, revise
Rationale Brief justification
Citation candidates Sources suitable for final citation
Final wording Calibrated wording for the report
Reviewer status Accepted, rejected, escalated

Thresholds vary by claim type: definitions need an authoritative source or explicit local definition; historical facts need a primary or high-quality secondary source; current facts need a fresh primary source or multiple recent reliable sources; comparative claims need evidence for each side; quantitative claims need an original dataset, benchmark, or documented methodology; causal claims need stronger evidence, mechanism, and caveats; recommendations need evidence plus explicit assumptions; safety/security claims need conservative wording and adversarial review. A claim passes only when cited evidence directly supports the wording; if evidence supports a weaker version, rewrite it; if evidence is contradictory, present the disagreement or move the claim to an appendix as uncertain.

Counter-Evidence Recall

A verified report demonstrates active search for counter-evidence — one of the most important differentiators between polished-but-fragile reports and genuinely reliable research. Dedicated contradiction agents for each major section and high-risk claim should ask: what would make this claim false? which credible sources disagree? are there benchmark limitations or replication failures? are there known incidents or negative results? is the claim only true under narrow assumptions? is the evidence stale? is the source financially or reputationally motivated? is the conclusion stronger than the evidence? are there alternative explanations? does a primary source contradict a secondary interpretation? Counter-evidence should be first-class state, addressed in the relevant sections, especially when it affects recommendations or confidence.

Agentic Tool-Use Loops

The system should implement explicit tool-use loops rather than one-shot calls: search → fetch → read → extract → verify → identify gaps → re-plan → repeat until thresholds are met or budget stops. Each loop is bounded by stop conditions:

Loop Stop Condition
Search expansion New results are redundant or low quality
Source reading Evidence coverage target met
Counter-evidence search Credible disagreement search exhausted within budget
Claim verification All high-risk claims pass, fail, or escalate
Citation audit Every citation target matches claim support
Draft revision No critical reviewer findings remain

Loops should not be infinite or purely model-directed; the graph enforces max iterations, budget ceilings, quality thresholds, and human escalation when uncertainty remains. Use deterministic tools where possible: citation URL existence checked programmatically, document hashes computed, duplicate detection via canonical URLs and fingerprints, table extraction via parsers, and schema validation rejecting malformed outputs before they enter the evidence graph.

Parallel Subagent Orchestration

Parallel subagents serve diversity, coverage, and independent verification. The strongest pattern is not "ask ten agents and merge" but role-specific fan-out with independent artifacts and cross-review:

Pattern Use
Query fan-out Different agents generate and execute search strategies
Source-class fan-out Agents specialize in academic, regulatory, vendor, incident, and critical sources
Section fan-out Agents research different report sections
Claim fan-out Agents verify different claims
Red-team fan-out Agents attack sources, citations, assumptions, and synthesis
Jurisdiction/language fan-out Agents cover regional or multilingual evidence
Time-window fan-out Agents compare historical versus current sources
Reviewer fan-out Independent reviewers audit the same high-risk claims

Fan-in is where many systems fail. The merge stage should not average opinions or concatenate summaries; it should deduplicate sources, normalize claims, resolve conflicts, rank evidence, and preserve dissent, with a lead synthesizer working from the evidence graph rather than raw worker prose. Subagent outputs should be schema-constrained — a search worker returns query records and source candidates, a reader returns evidence cards, a verifier returns claim decisions, a reviewer returns findings — preventing workers from smuggling unsupported prose into the report.

State, Memory, and Replay

Every report gets a run directory or checkpoint store, inspectable without the original chat session. A practical layout:

runs/
  2026-06-01-topic-slug/
    manifest.json
    config.yaml
    plan/ (research_plan.json, section_matrix.json, source_policy.json)
    search/ (queries.jsonl, results.jsonl)
    sources/ (source_cards.jsonl, raw/, normalized/, snapshots/)
    evidence/ (evidence_cards.jsonl, contradiction_cards.jsonl)
    claims/ (claim_cards.jsonl, verification_decisions.jsonl)
    drafts/ (outline.md, draft_v1.md, draft_v2.md)
    reviews/ (citation_audit.jsonl, adversarial_review.jsonl, human_review.jsonl)
    final/ (report.md, references.bib, provenance.json)
    logs/ (tool_calls.jsonl, model_calls.jsonl, budget.jsonl, errors.jsonl)

The manifest records model names, prompt versions, tool versions, search-provider configuration, run times, cache settings, policy settings, and seeds where applicable. Replay should support three modes: exact replay (reuse cached tool and model outputs to reproduce a run — supports audit), tool replay (reuse fetched sources but rerun model extraction and synthesis — supports prompt/model regression tests), and fresh replay (rerun search and fetch with the same plan, recording drift — supports freshness checks).

Source Integrity and Adversarial Robustness

A verified agent on the open web must assume some documents are low-quality, manipulative, generated, copied, stale, impersonated, or poisoned:

Threat Description Mitigation
SEO spam Pages optimized to rank while derivative or false Source triage, authority scoring, duplicate detection, primary-source preference
Poisoned documents Content crafted to manipulate AI extraction or instructions Treat web text as data, strip prompt-like instructions, isolate source content
Fabricated citations Nonexistent papers, broken references, fake DOIs Citation existence checks, DOI/title lookup, source-fetch validation
Source impersonation Lookalike domains, spoofed organizations Domain verification, organization cross-checks, TLS/canonical checks
Citation laundering Secondary source repeats unsupported claim Trace claims to primary evidence
Stale documentation Old pages contradict current behavior Freshness metadata, current-source requirements, date-aware synthesis
Benchmark gaming Claims from narrow or non-representative evaluations Benchmark-methodology extraction, caveat preservation
Hidden conflicts Vendor or affiliate incentives Source-type and independence scoring
Selective evidence Only pro-claim sources found Dedicated counter-evidence agents
Prompt injection in sources Source text instructs the agent to ignore rules Tool-output isolation and prompt-injection filters

A critical rule: fetched documents are untrusted input and must never issue instructions to the agent. The prompt must separate source content from system instructions, and extraction must ignore document text that attempts to direct the agent. Use a source-integrity score but never rely on a single scalar — preserve risk reasons such as suspicious domain, no author, copied content, no publication date, broken citations, generated tone, excessive affiliate links, or title/content mismatch.

Fabricated-citation defense

Citation hallucination is especially dangerous in long-form reports because fabricated references make unsupported claims look authoritative. The citation layer should enforce: every final citation corresponds to a fetched source record; every cited source has a stable identifier (URL, DOI, ISBN, archive link, repository commit, or document hash); every citation supports the sentence where it appears; quoted text matches the retrieved source; citation metadata is never invented (missing metadata stays missing); broken or inaccessible sources are flagged before finalization; secondary citations are not treated as primary evidence unless the original is unavailable and that limitation is disclosed; and bibliography entries are derived from source metadata, not model memory. A citation auditor should run after drafting because claims drift during synthesis.

Source-impersonation defense

Impersonation is broader than fake domains — a page can look like official documentation while being an outdated mirror, a copied tutorial, a sponsored post, or a malicious document. Validate identity using multiple signals: domain ownership and canonical links, organization cross-links, TLS and redirect path, publication venue, author identity, repository ownership, DOI/registry lookup, archive history, and cross-source consistency. For high-risk claims, prefer sources both authoritative and close to the original event, specification, dataset, or system: a blog about a standard is weaker than the standard; a news summary of a benchmark is weaker than the benchmark paper or repository; a vendor performance claim is weaker than an independently reproducible benchmark.

Human-in-the-Loop Governance

Human review should be reserved for decisions where automation is brittle, consequential, or value-laden — not as a substitute for basic verification:

Gate Human Decision
Scope approval Is the research question and outline correct?
Source policy approval Are allowed/excluded source classes appropriate?
Budget approval Is the planned spend/latency acceptable?
High-risk claim escalation Include, soften, or remove uncertain/disputed claims?
External publication approval Is the final report ready for distribution?
Policy exception approval Use a lower-quality source for lack of alternatives?

The system should support interactive and non-interactive modes. In interactive mode the user approves, revises, or rejects gates; in non-interactive mode, policy files define defaults ("drop uncertain claims," "cap cost at X," "include disputed claims only in an appendix"). Review artifacts are preserved as part of the run, so a report can say not merely that it was reviewed but which claims were escalated and how they were resolved.

Report Synthesis

Final synthesis is a constrained transformation from verified claims and evidence, not unconstrained essay generation. The synthesis stage should use the approved outline; pull only passed claims into main sections; preserve uncertainty and disagreement; prefer calibrated language over overclaiming; attach citations near the claims they support; avoid claims without evidence cards; move failed or uncertain claims to an appendix or omit them per policy; generate an evidence-to-section trace; and trigger a citation audit after each major revision. Readability should not override traceability: when a claim cannot be cited cleanly, weaken it, remove it, or mark it unresolved rather than hide the gap.

Operational Surface

A verified-research system should feel like a serious engineering tool, not a chatbot transcript. It should expose commands to initialize a project, generate a plan from a topic, run with a profile, resume and inspect a run, verify, render, benchmark, show cache statistics, and diff two runs. Useful profiles include:

Profile Behavior
quick Small fan-out, light verification, short report
standard Moderate fan-out, claim verification, citation audit
deep Broad fan-out, counter-evidence search, adversarial review
adversarial Heavy red-team checks and source-integrity review
archival Strong snapshotting and provenance preservation
offline Uses local caches and corpora only
benchmark Deterministic settings for evaluation

Run state should be inspectable at a glance, for example: plan approved; 184 sources discovered, 72 read, 38 accepted, 19 rejected as low integrity; 412 evidence cards; 146 candidate claims with 103 pass, 21 revised, 14 uncertain, 8 fail; citation audit passed, adversarial review with 3 warnings; budget and elapsed time within limits. The user should be able to inspect any final paragraph and trace it back to claims and evidence.

Data Model

A minimal data model includes: ResearchPlan, ResearchQuestion, SectionSpec, QueryRecord, SearchResult, SourceCard, DocumentSnapshot, EvidenceCard, ContradictionCard, ClaimCard, VerificationDecision, ReviewFinding, HumanGateDecision, BudgetEvent, ModelCallRecord, ToolCallRecord, ReportDraft, and FinalReport. Each object should have stable IDs, timestamps, provenance, and schema versions; schema evolution matters because long-running systems accumulate old runs that must remain readable. The claim card and evidence card are the most important primitives — if they are weak, the report will be weak regardless of orchestration sophistication.

Quality Gates

The system enforces quality gates before final rendering:

Gate Requirement
Source coverage Each major section has enough accepted sources
Primary-source coverage High-risk technical claims use primary sources where available
Claim verification Main-text claims are pass only
Counter-evidence High-risk claims have explicit contradiction search
Citation support Citations support nearby claim wording
Quote accuracy Quoted spans match source text
Source integrity No high-risk source used without disclosure or approval
Freshness Current claims use recent or canonical current sources
Budget compliance Run stays within configured limits or records approval
Reviewer closure Critical findings resolved before final

The system should fail closed: if a report cannot satisfy its policy, produce a partial report with unresolved claims separated, rather than silently downgrade verification.

Evaluation Benchmarks

Evaluation should measure the properties that matter for verified long-form research, not whether the report sounds good:

Dimension Metric
Factuality Fraction of atomic claims judged true or well-supported
Citation support Fraction of cited claims directly supported by the cited source
Citation existence Fraction of references that resolve to real sources
Quote accuracy Exact or faithful quote support
Counter-evidence recall Fraction of known opposing evidence surfaced
Source quality Share of claims supported by primary/high-authority sources
Calibration Whether uncertainty language matches evidence strength
Completeness Coverage of required research questions
Conflict handling Correct representation of disagreements
Robustness Performance under poisoned, spammy, or misleading sources
Reproducibility Ability to replay and obtain comparable artifacts
Cost efficiency Quality per dollar and per minute
Latency End-to-end and stage-level completion time
Human-review burden Number and severity of escalations
Regression stability Change in scores across prompt/model/tool updates

A benchmark suite should include synthetic tasks (for known ground truth and adversarial injections) and real tasks (for messy source ecosystems, stale pages, ambiguous evidence, and conflicting interpretations).

Factuality evaluation

Factuality evaluation should be claim-level — paragraph-level judgments are too coarse and hide unsupported assertions. A robust evaluator splits the report into atomic claims, maps each to citations and evidence, judges whether evidence directly supports the claim, checks for overstatement, identifies uncited factual claims, compares against known counter-evidence, and assigns severity. Error categories include unsupported, mis-cited, overstated, contradicted, stale, ambiguous, fabricated, misquoted, and missing-caveat — and the evaluator should produce actionable repair instructions (drop, revise, cite a stronger source, add a caveat, or move to an appendix).

Citation-support evaluation

Citation support is narrower than factuality: it asks whether the cited source supports the cited statement. A report can be factually correct but poorly cited, or well-cited to a source that is itself wrong. Checks: local support (does the citation support the sentence?), scope match (same scope, date, population, version, system?), strength match (wording no stronger than the source?), source-type fit, quote match, metadata validity, and link durability. The auditor should operate after final formatting because citations move during editing.

Counter-evidence evaluation

Counter-evidence recall should be benchmarked explicitly — a system that only finds supporting evidence is not doing verified research. Construct tasks with known disagreements, retractions, benchmark caveats, regulatory criticisms, replication failures, or outdated conventional wisdom, and measure whether the report found and represented these disconfirming sources. Scoring includes known-counter-source recall, counter-claim recall, integration quality (did disagreement affect conclusions?), caveat placement, an overconfidence penalty, and search-trace quality. This protects against citation-rich confirmation bias.

Adversarial evaluation

Adversarial benchmark tasks should include hostile source environments: SEO pages outranking primary sources; lookalike domains impersonating official docs; fabricated citations in plausible prose; prompt-injection text inside fetched documents; outdated documentation with current-looking titles; vendor claims contradicted by independent benchmarks; copied pages that strip caveats; forum claims repeated as facts; broken citation chains; conflicting sources with different definitions; rebranded projects with stale names; content farms summarizing each other; academic-looking PDFs with no legitimate venue; real papers cited for claims they do not support; and benchmark tables missing evaluation caveats. The system is scored on whether it detects, discounts, or escalates these cases.

Cost, Latency, and Budget Allocation

A production agent must treat cost and latency as design constraints. Budget allocation should be risk-based: low-risk background claims use accepted sources and light verification; high-risk factual claims use primary sources and independent verification; current claims spend on fresh search and validation; quantitative claims spend on original tables and method checks; controversial claims spend on counter-evidence and reviewer passes; repeated source domains are cached aggressively; redundant sources stop being read when marginal value drops; and a final citation-audit budget is always reserved. The planner estimates cost before execution; the scheduler updates expected remaining cost; and if the run approaches a limit, it degrades gracefully — reducing fan-out for low-risk sections, summarizing unresolved areas, or asking for approval. Latency is reduced by parallelism, caching, and pipelining (overlapping search, fetch, and read), though final synthesis should wait for verification gates.

Caching strategy

Cache layers: search-result cache (repeated queries within a time window), document cache (avoid refetching unchanged sources), normalization cache (cleaned HTML/PDF text), extraction cache (reuse evidence cards when source hash and extractor version match), embedding cache, verification cache (reuse claim-evidence decisions when both are unchanged), model-response cache (exact replay and cost control), citation-audit cache, and benchmark cache. Caches must be invalidated by content hash, tool version, prompt version, model version, and schema version — a stale cache can be more dangerous than no cache if outdated source content silently backs current claims. The system should support cache and freshness policies (off / read / write / read-write; current / allow-stale; exact replay).

Model-allocation strategy

A single frontier model should not do every task. Allocate by reasoning difficulty and risk:

Task Suggested Tier
Query expansion Cheap or medium model
Search-result triage Cheap or medium model plus rules
Document extraction Medium model
Table and metadata parsing Deterministic tools first
Claim verification Strong reasoning model
Counter-evidence synthesis Strong reasoning model for high-risk areas
Adversarial review Strong reasoning model
Final synthesis Strong reasoning model
Formatting Cheap model or deterministic renderer

A strong full-reasoning model is best used as the lead planner, high-risk verifier, adversarial reviewer, and final synthesizer — calling cheaper workers and deterministic tools rather than personally reading every low-value document. Escalation rules should be explicit: if a medium worker marks a claim high-impact, contradicted, quantitative, or current, route it to the strong verifier; if source-integrity risk is high, route to adversarial review.

Determinism and Reproducibility

LLM systems are not perfectly deterministic, but the surrounding architecture can make runs auditable and replayable through stable graph definitions, versioned prompts and schemas, recorded model names and parameters, cached model outputs for exact replay, source snapshots and hashes, deterministic merge logic where possible, explicit seeds for stochastic components, diff tools between runs, and regression tests over benchmark tasks. Distinguish reproducibility of artifacts (exact replay from cache) from reproducibility of live web conditions (fresh replay intentionally detects drift).

Governance and Safety

Governance should be embedded in the runtime, not treated as post-hoc policy text: policy files (source rules, budget caps, citation standards, human gates), audit logs, review queues, access controls (limit tools, credentials, external writes), a publication gate, PII and sensitive-data filters, external-state controls (no contacting people, posting, or purchasing without approval), license and terms awareness, and a disclosure policy stating limits and uncertainty. The system should support organizational profiles — legal, medical, security, or financial reports require stricter gates than a general technical overview.

Implementation Blueprint

A practical implementation can be staged.

Stage 1 — Minimal verified-report pipeline. Build commands for plan/run/resume/render, a run-directory manifest, schemas for research plan, source card, evidence card, and claim card, a search/fetch/read loop, a claim-verification pass, a citation-audit pass, and a Markdown renderer. Success: a short report where every main claim maps to evidence cards and final citations resolve.

Stage 2 — Parallel research graph. Add section-level and query-family fan-out, source-class specialists, fan-in deduplication, budget accounting, checkpointed resume, and model allocation by task. Success: a multi-section report with parallel workers that recovers from interrupted nodes.

Stage 3 — Adversarial robustness. Add source-integrity scoring, prompt-injection handling for fetched content, citation-existence validation, source-impersonation checks, counter-evidence agents, an adversarial reviewer, and disputed-claim escalation. Success: benchmark tasks with spam, poisoned documents, fake citations, and impersonation are detected or escalated.

Stage 4 — Evaluation harness. Add a claim-level factuality evaluator, a citation-support evaluator, a counter-evidence recall benchmark, cost/latency metrics, regression dashboards, and run-diff tooling. Success: prompt, model, and tool changes can be evaluated against stable benchmark suites.

Stage 5 — Production hardening. Add cache policies, archive/snapshot integration, a human-review workflow, policy profiles, CI integration, team run sharing, and security/credential isolation. Success: the system can be used repeatedly in real projects with predictable cost, inspectable provenance, and controlled publication.

Recommended Graph Specification

A concrete graph:

START
  -> intake.normalize
  -> plan.generate
  -> plan.review_gate
  -> search.expand_queries
  -> search.execute[*]
  -> sources.triage
  -> sources.integrity_screen
  -> fetch.retrieve[*]
  -> docs.normalize[*]
  -> evidence.extract[*]
  -> claims.generate
  -> counterevidence.plan
  -> counterevidence.search[*]
  -> counterevidence.extract[*]
  -> claims.verify[*]
  -> claims.rewrite_or_reject
  -> synthesis.outline
  -> synthesis.draft
  -> citations.audit
  -> adversarial.review
  -> human.final_gate
  -> render.final
END

Conditional edges:

if plan_review == rejected -> plan.generate
if source_integrity == high_risk -> source.reject_or_escalate
if claim_verification == fail -> appendix_or_drop
if claim_verification == uncertain -> counterevidence.search or human_gate
if citation_audit == fail -> synthesis.revise
if adversarial_review == critical -> claims.verify or source_integrity_screen
if budget_remaining  budget_gate

Parallel nodes should be keyed by section, query family, source, document, and claim, allowing fine-grained retries and scalable execution.

Production Failure Modes

Failure Mode Symptom Prevention
Polished unsupported prose Reads well but claims lack support Claim-card gate before synthesis
Citation drift Citation supports nearby but not the exact claim Post-draft citation audit
Source over-trust Weak source treated as authoritative Source hierarchy and integrity scoring
Confirmation bias Only supporting sources included Counter-evidence agents
Cost runaway Repeated loops without convergence Budget ceilings and stopping conditions
Stale cache Old source used for current claim Freshness policy and hash invalidation
Worker hallucination Subagent invents source details Fetch-backed source records only
Merge loss Fan-in discards minority evidence Contradiction cards and conflict-preserving merge
Hidden prompt injection Source text manipulates the agent Tool-output isolation
Human rubber-stamp Review gate too vague Claim-specific escalations

The system should make failure visible: a partial but honest report is preferable to a complete but unsupported one.

Engineering Recommendations

  1. Use graph-native orchestration with checkpointed typed state as the core abstraction.
  2. Treat every web document as untrusted data.
  3. Make source cards, evidence cards, and claim cards mandatory.
  4. Require direct claim-to-evidence mapping before final prose.
  5. Use parallel workers for discovery and extraction, but strong centralized verification for high-risk claims.
  6. Preserve counter-evidence as first-class state.
  7. Enforce citation-existence and citation-support checks after drafting.
  8. Use model tiers instead of a single expensive model for every node.
  9. Cache by content hash, prompt version, model version, tool version, and schema version.
  10. Support exact, tool, and fresh replay.
  11. Add human gates only where they materially improve judgment, governance, or safety.
  12. Benchmark factuality, citation support, counter-evidence recall, adversarial robustness, cost, and latency.
  13. Fail closed when claims do not pass verification.
  14. Keep final report generation downstream of verified artifacts.
  15. Build inspectability into the tool from the first version.

Reference Component Map

A verified-research runtime can be organized into these components:

Component Responsibility
Frontend Commands, configuration, run management
Graph runtime Node execution, edges, checkpoints, retries
State store Typed artifacts, logs, run manifest
Search adapters Web, academic, local, custom search providers
Fetch adapters HTTP, browser, PDF, archive, repository fetch
Normalizers HTML/PDF/text/table cleanup
Source-integrity module Authority, spam, impersonation, prompt-injection checks
Evidence extractor Source-to-evidence transformation
Claim engine Claim generation, deduplication, verification
Counter-evidence engine Disconfirmation planning and search
Synthesis engine Outline and drafting from passed claims
Citation auditor Citation existence and support validation
Review engine Adversarial and human review gates
Evaluation harness Benchmarks, scoring, regression tests
Cache manager Search, source, extraction, verification, model caches
Renderer Markdown, PDF, provenance bundle

The system should ship with policy profiles, schemas, report templates, benchmark tasks, and examples. The default mode should prioritize correctness over speed, while allowing explicit profiles for quick exploratory reports.

Evidence and Claim Handling Policy

Main-report synthesis should use only claims that pass verification. A passed claim means: it is atomic enough to evaluate; source evidence directly supports the wording; the citation target exists and was fetched or validated; known credible counter-evidence was considered; the claim is not stronger than the evidence; source-integrity risks are acceptable or disclosed; and any required reviewer gate has accepted it. Recommended verification labels:

Label Meaning
PASS Supported strongly enough for the main report
PASS_WITH_CAVEAT Supported only with explicit limitation
REVISE Evidence supports weaker or narrower wording
UNCERTAIN Evidence is mixed, insufficient, or ambiguous
FAIL Unsupported, contradicted, fabricated, stale, or mis-cited
ESCALATE Human or stronger-model review required

Claims that fail or remain uncertain should not appear as factual main-text assertions; they may be omitted, rewritten, or placed in an appendix with status and reason.

Conclusion

A verified deep-research system can exceed ordinary human research-team performance if it is designed as a graph-driven evidence machine rather than a conversational writing assistant. The core primitives are durable graph state, parallel subagents, typed source and evidence artifacts, claim-level verification, adversarial source-integrity checks, counter-evidence recall, citation auditing, human governance gates, and budget-aware production execution. The essential principle is simple: final prose is the last step, not the research substrate. The system should first build a verified evidence graph, then synthesize only passed claims; everything else should be rejected, revised, escalated, or moved to an appendix. This discipline is what turns large-context reasoning, massive parallel search, deterministic replay, and programmatic tool composition into a research architecture meaningfully stronger than either a single AI agent or an unaudited human team.

Sources