AlienMachine CMS

Chain, Tree, and Graph of Thoughts: How Structured Reasoning Maps onto Agent Frameworks

Three "structure-enhanced" prompting paradigms dominate the 2022–2024 literature on guiding LLM reasoning with an explicit topology: Chain-of-Thought (a linear chain), Tree-of-Thoughts (a search tree), and Graph-of-Thoughts (an arbitrary directed graph). They form a clean containment hierarchy and each has real benchmark wins — but those wins are narrow, bound to a single task/model/metric, and measured against single-sample baselines. The practical upshot for 2026: reach for a heavier topology only when your task decomposes cleanly, you have a cheap verifier, and your model is not already a strong internal reasoner. Otherwise the simplest workflow that works is usually cheaper, faster, and easier to maintain.

Every quantitative claim below is stated with its task, model, and metric, because the most common error in this literature is generalizing a benchmark-specific number. The Limitations section records the strongest objections an informed skeptic would raise; if you are making a production decision, read it before the performance tables.

The three paradigms at a glance

  • Chain-of-Thought (CoT) — a linear chain of intermediate reasoning steps (Wei et al., 2022).
  • Tree-of-Thoughts (ToT) — a tree search over partial-solution states with an LLM generator, an LLM evaluator, and BFS/DFS with backtracking (Yao et al., 2023).
  • Graph-of-Thoughts (GoT) — an arbitrary directed graph that adds two operations trees cannot express: aggregation (merging multiple thoughts) and refinement (a self-loop that improves a thought in place) (Besta et al., 2024).

Besta et al.'s survey Demystifying Chains, Trees, and Graphs of Thoughts unifies these as reasoning topologies and shows the structural containment chain ⊆ tree ⊆ graph.

The benchmark evidence is real but narrow. ToT raises Game-of-24 success from 4.0% (CoT) to 74% (ToT, b=5) on GPT-4. GoT improves sorting quality by ≈62% over ToT while cutting its cost metric by >31%, on GPT-3.5, at the largest tested problem size (P=128). Crucially, these two headline numbers are not comparable to each other — different models, tasks, and metrics — and both are measured against single-sample baselines rather than compute-matched ones.

The three reasoning topologies map by analogy onto today's agent-orchestration frameworks — LangGraph (explicit stateful graphs), AutoGen (multi-agent conversation), and Anthropic's five workflow patterns — but no primary source asserts these equivalences, and the frameworks supply the orchestration plumbing, not the LLM-based evaluators that do the actual reasoning work. Anthropic's own guidance counsels simplicity first, which is itself evidence against reaching for heavy topologies by default.

1. Background

Research question

How do Chain-of-Thought, Tree-of-Thoughts, and Graph-of-Thoughts compare in design, performance, and cost, and how do these reasoning structures map onto production agent-orchestration frameworks (LangGraph, AutoGen) and Anthropic's agent workflow patterns?

A note on terminology

A reasoning topology is the class of graph G = (V, E) used to model the reasoning process: vertices are "thoughts" (coherent intermediate language steps) and edges are dependencies between them. Plain input→output prompting is a single-node graph; CoT is a path; ToT is a tree; GoT is an arbitrary directed graph. The discriminating structural axis is the pair of operations a topology admits: branching (out-degree > 1) and aggregation (in-degree > 1).

2. Design and structure

Paradigm Topology Distinguishing operations Control machinery Primary source
IO prompting single node none — Yao et al.; Besta et al.
CoT linear chain sequential generation none Wei et al. 2022
CoT-SC (Self-Consistency) k parallel chains + vote branching (independent) + majority vote argmax_y #{i : yᵢ = y} Yao et al.; Besta et al.
ToT tree branching + backtracking search generator G, evaluator V, BFS/DFS Yao et al. 2023
GoT arbitrary directed graph branching + aggregation + refinement Prompter / Parser / Scorer / Controller (GoO + GRS) Besta et al. 2024

CoT introduces intermediate thoughts z₁…zₙ bridging input and answer, sampled as one continuous sequence. CoT-SC ensembles k i.i.d. chains and returns the most frequent answer — richer sampling, but with no exploration within a chain.

ToT frames the problem as search over a tree whose nodes are partial states s = [x, z₁…ᵢ]. A ToT instance is defined by four choices: (1) thought decomposition; (2) a generator G(p_θ, s, k); (3) a state evaluator V(p_θ, S) (independent scalar value, or vote across states); (4) a search algorithm — BFS (keep the b best states per step; used for Game-of-24 and creative writing) or DFS (explore most-promising first, prune below a value threshold, backtrack; used for crosswords). IO, CoT, CoT-SC, and self-refinement are all special cases of ToT.

GoT models reasoning as an arbitrary directed graph and is the only scheme of the four that supports arbitrary-graph (aggregation) structure. It defines three transformation classes:

  • Aggregation — a vertex with multiple incoming edges merges several thoughts/chains into one (e.g., combining partial solutions, or merging documents);
  • Refining — a self-loop edge (v, v) that improves a thought in place;
  • Generation — one thought spawns k children (this subsumes the ToT/CoT-SC expansion step).

GoT's runtime is organized around a Graph of Operations (GoO) — a static decomposition of the task into transformations and their dependencies — and a dynamic Graph Reasoning State (GRS), driven by four modules: Prompter, Parser, Scoring module, and Controller.

Besta et al.'s survey generalizes all of this into a blueprint decomposing any scheme along: topology class (chain/tree/graph), topology scope (single- vs multi-prompt), representation (implicit vs explicit), derivation (manual / LLM-automatic / semi-automatic), plus the reasoning schedule.

Caveat (structural, not performance). The containment CoT ⊆ ToT ⊆ GoT is an expressivity relationship — a chain is a degenerate tree, a tree a degenerate DAG. It does not imply that a richer topology reasons better on every task; greater structure raises the capability ceiling but also adds cost and new failure surfaces. The containment framing originates with GoT's own authors.

3. Performance

All figures below are within-paper comparisons (same model, same task, same paper's baseline). They are not comparable across papers — see the model-confound caveat at the end of this section.

Tree-of-Thoughts (GPT-4)

Task Metric CoT ToT Notes
Game of 24 success rate 4.0% 74% (b=5) IO 7.3%; CoT-SC(k=100) 9.0%; ToT(b=1) 45%
Creative Writing coherency (1–10) 6.93 7.56 IO 6.19; humans preferred ToT 41/100 vs CoT 21/100
Mini-Crosswords word-level success 15.6% 60% game-level: CoT 1% → ToT 20% (4/20); ablations show pruning + backtracking both matter

The Game-of-24 result is ToT's flagship: roughly 60% of CoT samples failed after the very first step, which a search-with-backtracking topology directly remedies.

Caveats (scope + budget). (i) "74%" is Game-of-24 only; ToT's gains on its other two tasks are smaller and measured on different metrics. (ii) The 4% CoT baseline is single-sample greedy; ToT spends many more model calls, so part of the gap is "search + verifier + more compute," not topology alone. A compute-matched CoT-SC baseline narrows it. (iii) Game-of-24 has a cheap exact verifier (does the expression equal 24?) — a luxury most production tasks lack.

Graph-of-Thoughts (GPT-3.5)

On the sorting task, GoT improves quality by ≈62% over ToT and by ≈70% over CoT. The precise headline — "62%" — is the median-error reduction at problem size P=128 (the largest tested), and the advantage is size-dependent: negligible at P=32, ≈61% at P=64, ≈69% at P=128 vs the GoT paper's ToT baseline. GoT was additionally evaluated on set intersection, keyword counting, and document (NDA) merging, improving over ToT across instances.

Caveats. (i) "62%" is sorting-specific and size-specific (median error at P=128); it does not transfer uniformly to other tasks. (ii) The "ToT" in this comparison is the GoT authors' own re-implementation of ToT on sorting, not Yao et al.'s published ToT — do not conflate it with the 74% Game-of-24 result. (iii) These tasks (sorting, set-ops, counting) have a literal decompose-solve-aggregate shape that GoT's aggregation operation was built to exploit, and are themselves poor real-world LLM use-cases.

The cross-paper comparability problem

ToT's results use GPT-4 (experiments mid-2023); GoT's results use ChatGPT / GPT-3.5. Therefore:

  • Valid: within-paper deltas — "CoT 4% → ToT 74% on GPT-4," "GoT 62% over ToT on GPT-3.5 sorting."
  • Invalid: any absolute ranking such as "GoT > ToT > CoT" assembled by lining up 74% next to 62%. The numbers measure different models on different tasks with different metrics.

4. Cost and efficiency

Paradigm LLM-call scaling Cost class
CoT ~O(1) chain (one pass) cheapest
CoT-SC ×k samples linear in k
ToT grows with branch factor × depth (+ an evaluation call per node) high
GoT adds aggregation/refine calls; beats ToT cost metric on suitable tasks high, but 31% cost reduction over ToT** on sorting at P=128 while improving quality. Both ToT and GoT cost far more than IO/CoT, driven by generating k new thoughts per expansion — the same multiplicity that produces their quality gains.

GoT's cost theory: volume and latency

GoT's central efficiency argument is a structural tradeoff between two graph-theoretic metrics:

  • Latency = number of hops (longest path) from input to a final thought;
  • Volume = number of preceding thoughts that can influence a given thought (its information reach).
Scheme Latency Volume
CoT N N
CoT-SC N/k N/k
ToT log_k N O(log_k N)
GoT log_k N N

GoT is the only scheme achieving high volume and low latency — every final thought can be influenced by all N prior thoughts (via aggregation) while keeping the path short.

Caveat (units). Latency and volume here are graph metrics, not wall-clock seconds or token counts. The "31% cost reduction" is computed in this synthetic cost model. It does not translate directly to a 31% smaller API bill, and GoT often issues more absolute LLM calls than ToT in some configurations. Anthropic frames the practical reality bluntly: agentic systems "trade latency and cost for better task performance," and complexity should be added only when it demonstrably improves outcomes.

5. Mapping to agent-orchestration frameworks

This section is analytical, not documented fact. No primary source — not Anthropic, LangGraph, AutoGen, nor the CoT/ToT/GoT papers — asserts these correspondences. The frameworks do not claim to "implement GoT." The mappings below are interpretive aids; they are also many-to-many, not 1:1. The underlying framework primitives are individually documented; the cross-mapping is an interpretation.

LangGraph — the explicit stateful graph

LangGraph models a workflow as a graph of three documented primitives: State (a shared, typed schema), Nodes (functions that read state and return updates), and Edges (fixed transitions or conditional branches). Cycles are first-class; branching uses fan-out/fan-in. Execution follows a Pregel-style super-step message-passing model. A checkpointer (e.g., MemorySaver, SqliteSaver) snapshots state at every step, enabling persistence, time-travel debugging, fault tolerance, and human-in-the-loop via interrupt()/Command. Subgraphs compose over a shared state key. LangChain v1's create_agent is built on LangGraph and inherits these properties.

AutoGen — multi-agent conversation

AutoGen's high-level AgentChat API centers on conversing agents: AssistantAgent (LLM-backed) and UserProxyAgent (solicits human input and/or executes code), orchestrated for multi-agent work via GroupChat + a GroupChatManager that decides who speaks next. A version split matters in 2026: 0.2 is the stable, actor-model-free branch; 0.4+ is a ground-up redesign on an event-driven actor model, split into autogen-core, autogen-agentchat, and extensions (with Microsoft also steering users toward a successor Agent Framework via a migration guide). Where LangGraph is graph-centric (orchestration is the declared topology), AutoGen is conversation-centric (orchestration emerges from message passing).

Anthropic — five workflow patterns

Anthropic's Building Effective Agents distinguishes workflows (LLMs orchestrated through predefined code paths) from agents (LLMs that dynamically direct their own process), atop an "augmented LLM" building block. The five workflow patterns:

  1. Prompt chaining — fixed sequence of calls, each consuming the prior output (optional programmatic gates).
  2. Routing — classify input, dispatch to a specialized follow-up (incl. cheap-vs-capable model routing).
  3. Parallelization — sectioning (independent subtasks) and voting (same task, multiple runs).
  4. Orchestrator-workers — a central LLM dynamically decomposes a task, delegates to workers, and synthesizes results (subtasks are not predefined).
  5. Evaluator-optimizer — a generator and an evaluator loop until criteria are met.

The paper's overriding counsel: find the simplest solution; add complexity only when it demonstrably improves outcomes; favor transparency and a well-crafted agent-computer interface.

The correspondence (interpretive)

Reasoning topology LangGraph echo AutoGen echo Anthropic pattern echo
CoT (linear chain) sequence of nodes single agent's stepwise turn Prompt chaining
CoT-SC (parallel + vote) fan-out nodes + reducer multi-agent debate Parallelization (voting)
ToT (branch + search + backtrack) conditional edges + back-edges + checkpoint replay GroupChat with a routing manager Routing + Evaluator-optimizer loop
GoT (branch + aggregate + refine) parallel branches → aggregator node; cyclic refine GroupChat that proposes/critiques/merges Orchestrator-workers (dynamic decompose + synthesize) + Evaluator-optimizer (refine)

Caveats (why this is analogy, not equivalence). (i) A LangGraph aggregator node is control-flow plumbing; GoT "aggregation" is an LLM reasoning operation that synthesizes thought content — same shape, different mechanism. (ii) AutoGen GroupChat is conversation, not value-pruned tree search. (iii) Critically, the frameworks give you the orchestration but not the LLM evaluator/value-model that does ToT/GoT's heavy lifting — the expensive, accuracy-critical part is left to you. (iv) Anthropic's framing is, if anything, evidence against defaulting to heavy topologies.

6. Critical limitations and honest caveats

This section records the strongest legitimate objections to this article's own thesis. None invalidate the cited facts; all bound their interpretation.

  1. Compute-matching is the missing control. None of the headline deltas (4%→74%, 62%/31%) are clearly normalized for LLM-call budget. ToT and GoT spend more compute than their single-sample baselines; part of every reported gain is "more sampling + a verifier," not topology per se. Each figure deserves a vs. compute-matched baseline qualifier.

  2. "Cost" is equivocal. The papers measure cost as synthetic volume/latency graph metrics. Production cost is dollars, p99 latency, and engineering/maintenance burden — for which these metrics are at best a loose proxy.

  3. Task selection favors the method. GoT's wins are on sorting/set-ops/counting/merging — tasks with an explicit aggregation seam, and tasks one would rarely hand to an LLM at all (sorted() is exact, free, instant). ToT's 74% is a small arithmetic search with a cheap oracle. External validity to open-ended reasoning is unestablished.

  4. Expressivity ≠ performance; "more structure = better" is non-monotonic. Containment guarantees representational power, not results. On simple tasks the topology overhead injects cost and error with no upside.

  5. Proponent-homogeneous evidence base. All three topology papers are authored by their proponents, and the integrating survey is by the GoT authors — a self-assessment conflict. No independent head-to-head meta-analysis is cited.

  6. Temporal scope (the strongest objection for 2026). CoT/ToT/GoT are 2022–2024 results, predating RL-trained long-reasoning models. By 2026, frontier models that internalize extended reasoning absorb much of what explicit prompt-topologies were bolted on to provide — doing their own branching/backtracking inside a single call, often cheaper to operate and maintain than a hand-built scaffold. Presenting these topologies as the production frontier would be citing a paradigm the field has partly moved past.

  7. CoT primary text was secondary-sourced. Wei et al. 2022 was cited by URL but not retrieved in full; CoT's characterization here rests on the consistent secondary descriptions in the survey and the ToT/GoT papers. The one first-hand CoT number used (4% Game-of-24) is measured within the ToT paper's own experiments, so it is direct.

Conclusion

The CoT → ToT → GoT progression is a coherent and genuinely useful map of structural options for guiding LLM reasoning: a chain when steps are linear, a tree when you must explore and backtrack against a verifier, a graph when partial results must be decomposed and recombined. The benchmark evidence for each is real but narrow, within-paper, and proponent-reported; the dramatic headline numbers (4%→74%, 62%/31%) are tightly bound to a single task, model, and metric and must not be generalized into an absolute "more structure wins" law.

For practitioners, the operative questions are not "which topology is best?" but: Does my task decompose? Do I have a cheap verifier? Is my model already a strong internal reasoner? When the answer to the first two is yes and the third is no, ToT/GoT-style structure earns its cost. Otherwise — which is most production work in 2026 — Anthropic's counsel holds: use the simplest workflow that works, and add structure only when it demonstrably pays. Agent-orchestration frameworks (LangGraph, AutoGen) are the right substrate for building any of these topologies, but they supply the graph plumbing, not the evaluator that makes structured reasoning actually reason.

Sources

  1. Besta, M., Memedi, F., Zhang, Z., Gerstenberger, R., et al. (ETH Zurich). Demystifying Chains, Trees, and Graphs of Thoughts. arXiv:2401.14295, 2024. https://arxiv.org/abs/2401.14295
  2. Besta, M., Blach, N., Kubíček, A., et al. (ETH Zurich). Graph of Thoughts: Solving Elaborate Problems with Large Language Models. AAAI 2024; arXiv:2308.09687. https://arxiv.org/abs/2308.09687
  3. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., Narasimhan, K. (Princeton / Google DeepMind). Tree of Thoughts: Deliberate Problem Solving with Large Language Models. NeurIPS 2023; arXiv:2305.10601. https://arxiv.org/abs/2305.10601
  4. Schluntz, E. & Zhang, B. (Anthropic). Building Effective Agents. 2024. https://www.anthropic.com/research/building-effective-agents
  5. Wei, J., Wang, X., Schuurmans, D., Bosma, M., et al. (Google). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022; arXiv:2201.11903. https://arxiv.org/abs/2201.11903
  6. LangChain. LangGraph documentation — Graph API, Persistence, Human-in-the-loop. https://docs.langchain.com/oss/python/langgraph/graph-api
  7. Microsoft. AutoGen — AgentChat User Guide and repository. https://microsoft.github.io/autogen/ · https://github.com/microsoft/autogen