AlienMachine CMS

Adaptive Context Pruning for Agentic AI: Techniques, Algorithms, and Implementation

Long-running agentic AI systems exhaust even million-token context windows, and naive truncation destroys the state that matters most: tool-call dependencies, early user intent, and exact constraints buried in file contents. The reliable answer is adaptive, structure-aware pruning that treats context as a dependency graph rather than a linear tape. The three highest-leverage moves are a dependency-graph pruner that preserves ancestry chains, a hybrid hot/warm/cold tier system with proactive token-budget triggers, and delta/diff-based file caching that replaces full re-reads with minimal patches.

Modern agentic systems such as Claude Code, OpenAI Codex CLI, and Cursor Agent Mode sustain sessions where file reads, tool outputs, and conversation history accumulate without bound. Static strategies (FIFO eviction, fixed summarization) discard critical state indiscriminately. This article synthesizes current techniques, presents concrete algorithm designs, and closes with a phased roadmap for building an adaptive context pruner.

The State of the Art

Two tracks of context compression

The field has split into two complementary tracks: model-side compression (KV-cache eviction, quantization, token dropping) and agent-side memory management (hierarchical storage, structure-aware eviction, task-aware pruning). For tool-augmented agents, agent-side approaches dominate, because model-side methods operate below the level of tool-call semantics and cannot distinguish a critical git diff from an expendable ls output.

Several research directions define the current frontier:

  • Adaptive memory-structure selection. Instead of committing to a single memory schema (vector RAG, graph, or flat buffer), an agent can carry multiple complementary structures and select among them with a learned gate, improving recall on long-context personalization benchmarks.
  • Intra-trajectory compression. An agent decides when to consolidate learnings into a persistent "knowledge" block and actively deletes raw interaction history, reducing total tokens while holding task accuracy roughly constant.
  • Tiered memory directories. Memory is organized into primary cache, secondary buffer, and a global archive, with redistribution driven by task or persona, reaching strong results on long-memory evaluation suites.
  • Multi-level hierarchies. Building a hierarchy (messages → episodes → semantics → themes) and retrieving top-down, expanding to raw messages only when reader uncertainty remains high, can substantially reduce tokens per query.
  • Task-aware line-level pruning for coding agents. A lightweight neural "skimmer" scores context lines against a goal hint generated by the agent, cutting tokens on coding benchmarks while aiming to hold or improve success rate.

Hierarchical memory architectures

Hierarchical memory is now the default design pattern. The consensus is a three-to-four tier stack:

Tier Name Contents Eviction Policy
Hot Working Context Recent turns, active tool outputs, current file buffers Recency-biased; never evict active tool chain
Warm Summarized Segments Compressed episode summaries, key decisions, user preferences Semantic relevance + dependency centrality
Cold Reference Archive Full interaction logs, file snapshots, historical tool outputs Retrieved on-demand; stored externally
Meta Index & Schema Dependency graph, entity registry, token budget ledger Persistent; tiny footprint

A recurring finding is that static tier sizes fail: the optimal split depends on task type (coding vs. research), interaction style, and current query complexity. Persona-driven redistribution and sparsity-versus-semantics objectives are two ways to make tier sizing adaptive.

Selective attention and relevance scoring

Relevance scoring has converged on multi-signal fusion:

  • Recency: exponential decay with a configurable half-life.
  • Semantic similarity: dense embedding distance to the current query.
  • Dependency centrality: PageRank or Katz centrality within the tool-call dependency graph. Messages that support many downstream actions score higher.
  • User importance markers: explicit signals (the user said "remember this" or pinned a file) override automatic scores.
  • Tool-chain criticality: tool results that are prerequisites for pending tool calls are immovable.

Relation-aware cache eviction policies formalize this by scoring each entry as a product of topic-level prevalence and topic structural importance derived from intra-topic dependency graphs, with the aim of improving cache hit ratios over LRU baselines.

Eviction policies beyond LRU/FIFO

Several alternatives to naive eviction have emerged:

  • Semantic-temporal aware eviction: ranks messages by a weighted combination of semantic redundancy (proximity to the conversation centroid in embedding space) and normalized age, aiming to preserve more factual "needles" than FIFO in needle-in-haystack tests.
  • Dependency-aware invalidation: classifies tools as READ or WRITE; a WRITE execution evicts stale READ caches by prefix-matching affected keys, reducing stale-read errors.
  • Ancestry extraction: instead of keeping the full trace, the system extracts the minimal dependency subgraph (root-to-leaf paths) required for the current reasoning step and compresses everything else into lightweight placeholders.

Context Pruning Algorithms and Techniques

Token-budget-aware summarization with controllable ratios

Token-budget summarization is the baseline: when context exceeds a threshold, older spans are compressed. The refinement is controllable compression ratios that vary by content type:

  • Tool reasoning chains: aggressive summarization (retain only conclusion plus key parameters).
  • File contents: moderate compression (structural outlines plus modified regions).
  • User directives: minimal compression (verbatim or lightly paraphrased).

A "sawtooth" pattern is the canonical implementation: the agent periodically consolidates a span into a knowledge block, then deletes the raw history. The compression ratio can be learned from task-completion feedback rather than fixed.

Sliding window with semantic anchors

A pure sliding window drops everything outside the last N turns. The semantic anchor extension preserves "keystone" messages across the boundary:

  1. Identify anchor candidates: user goals, system or tool configuration changes, major file edits, explicit preferences.
  2. Score candidates by composite relevance (recency + semantic centrality + dependency count).
  3. Retain top-K anchors in a pinned buffer that sits outside the sliding window.
  4. When the window slides, append summaries of evicted turns to a rolling tail summary rather than discarding them.

Adding linguistic-structure indexing — dependency parsing, coreference resolution, and discourse-relation tagging — to anchor selection has been shown to improve factual recall over standard retrieval-augmented baselines (see Semantic Anchoring in Agentic Memory, arXiv:2508.12630).

Graph-based context dependency tracking

The dependency graph is the dominant structural abstraction. A representative design is a DAG of reasoning steps where each node is a Thought, Action, or Observation and edges represent causal or logical prerequisites. Before each agent step, the system performs ancestry extraction — pulling the full dependency subgraph required for the current reasoning — then prunes validated but non-essential branches. Tool-level invalidation extends this: WRITE tools carry invalidation rules that map arguments to READ cache keys, forming a bipartite dependency graph between tool calls and cached results.

Relevance scoring models

The consensus relevance score is a weighted sum of normalized signals:

R(msg) = w_recency × Recency(msg)
       + w_semantic × Sim(embed(msg), embed(current_query))
       + w_centrality × Centrality(msg, dep_graph)
       + w_criticality × IsOnCriticalPath(msg, pending_tools)
       + w_user × UserImportance(msg)

Weights are tuned per task domain. For coding agents, w_criticality is highest (tool chains break if intermediate results vanish). For research agents, w_semantic dominates.

Hierarchical summarization

Hierarchical summarization builds a temporal pyramid:

  • Level 0: raw conversation turns.
  • Level 1: episode segments (e.g., "fixed auth bug in login.py") generated every N turns or at natural boundaries.
  • Level 2: episode summaries compressed into theme-level narratives.
  • Level 3: meta-summary of themes and open decisions.

Automating split/merge via a sparsity-versus-semantics objective prevents over-segmentation.

Delta/diff-based file updates

Re-reading an unchanged file is the single largest avoidable token cost in agentic sessions. The middleware pattern: intercept file reads, store compressed content keyed by hash, and return a short symbolic reference for unchanged files or a unified diff for changed ones. AST-aware compression and multiple read modes (diff, map, signatures) can reduce a large file from tens of thousands of tokens to a few hundred.

This is straightforward to implement: maintain a file_cache mapping paths to SHA-256 hashes and compressed contents. On re-read, hash the file; if unchanged, emit a symbolic reference; if changed, emit a unified diff.

Selective detachment of tool outputs vs. reasoning chains

Tool outputs and reasoning chains have different lifetimes:

  • Tool outputs (observations) are often bulky and quickly stale. Detach them after the dependent reasoning step completes, replacing them with a ToolResultSummary.
  • Tool reasoning chains (the thought → action → observation sequence) may be needed for debugging or for later steps that reference the same tool pattern. Compress them into ToolPattern entries.

Model-driven cache eviction for ReAct loops can achieve large peak token reductions by pruning stale tool outputs while preserving reasoning structure.

Implementation Strategies

Introspecting conversation and tool-call history

Modern LLM APIs provide the primitives needed for accurate pruning:

  • Token counting endpoints count input tokens including message history, system prompts, tool definitions, and tool_use/tool_result blocks without calling the model. (Anthropic: Count tokens in a Message.)
  • Context editing controls clear oldest tool results above a threshold and manage extended-thinking blocks. (Anthropic: Context editing.)
  • Inline budget awareness lets a model receive its remaining token budget and self-regulate.

The practical approach for a skill or agent:

  1. Pre-flight token audit: before each significant operation, count tokens on the projected message array (including pending tool results) to determine remaining budget.
  2. Tool-call logging: maintain a tool_log that records every tool_use/tool_result pair with metadata (tool name, arguments, result size, timestamp, result hash).
  3. Conversation index: build a running index of turn boundaries, tool-call spans, and file-read events. This index is tiny (under ~1K tokens) and never pruned.

Measuring token usage accurately

Do not estimate tokens with heuristics; use the API's count endpoint with the full message array, tools, and system prompt, then subtract from the model's window size to get remaining budget. Establish thresholds for proactive pruning:

  • Yellow zone: 70% of the window. Trigger warm-tier summarization.
  • Orange zone: 85% of the window. Trigger cold-tier archival and aggressive tool-output detachment.
  • Red zone: 95% of the window. Emergency semantic eviction; preserve only hot context plus anchors.

Maintaining continuity across pruning events

A user's mental model breaks when pruning removes information they believe the agent still has. Mitigations:

  • Pruning notifications: emit a brief message when context is pruned ("Consolidating earlier conversation into a summary...").
  • Anchor preservation: never prune user-stated goals, explicit constraints, or pinned references without confirmation.
  • Idempotent operations: design tool calls so that repeating them after pruning is safe (re-read a file rather than assuming its content is still in context).
  • Summary transparency: keep summaries human-readable, not opaque embeddings.

Handling file-content caching

Use a three-state model for file content in context:

State When to Use Representation in Context
Full text File actively being edited; small files; first read in session Verbatim file content
Symbolic reference File unchanged since last read; large stable files FileRef(path="src/auth.py", hash="a3f2...", status="unchanged")
Summary/outline File relevant but not active; API surfaces; dependency headers AST-derived signature map or section outline

A token-counting endpoint can verify that symbolic references save tokens (a FileRef is roughly 15 tokens versus thousands for source code).

Trigger conditions and integration

  • Proactive triggers: yellow/orange thresholds, periodic consolidation boundaries (e.g., every 20 turns), natural breakpoints (completion of a tool chain or file edit).
  • Reactive triggers: a context-window warning from the API, a tool result that exceeds remaining budget, or a new high-level goal that invalidates prior working context.

For million-token windows, proactive triggers are essential; reactive pruning at 95% leaves insufficient headroom for a multi-file tool chain. Integration touch points include a short system instruction describing memory state ("Working on auth refactor; 3 files active; 12 prior turns summarized"), a per-skill budget hint for long-running operations, and writing cold-tier summaries to persistent project memory rather than keeping them in the conversation.

Concrete Algorithm Designs

Algorithm A: Dependency-graph-based pruner

Purpose: preserve tool-call → file → user-query relationships; never prune a message whose descendant is still active.

Data structures:

class ContextNode:
    id: str
    type: Literal["user", "assistant", "tool_use", "tool_result", "file_read", "summary"]
    tokens: int
    content_hash: str
    timestamp: float
    dependencies: Set[str]   # node IDs this node depends on
    dependents: Set[str]     # node IDs that depend on this node
    embedding: List[float]
    user_importance: float   # 0.0–1.0
    status: Literal["hot", "warm", "cold"]

class DependencyGraph:
    nodes: Dict[str, ContextNode]
    edges: List[Tuple[str, str]]  # (source, target)

Logic:

  1. Ingest turn: after each model turn or tool call, create nodes and edges. User query → root. Assistant thought → depends on user query plus retrieved summaries. Tool use → depends on assistant thought. Tool result → depends on tool use. File read → depends on the triggering tool use or thought.
  2. Critical-path detection: for each pending tool call or open request, compute the dependency subgraph (all ancestors) and mark those nodes critical.
  3. Relevance scoring (non-critical nodes): score = 0.3·recency + 0.3·semantic + 0.2·centrality + 0.2·user_importance.
  4. Eviction when over threshold: (1) detach tool results for completed chains, replacing with summaries; (2) archive lowest-score leaf nodes to cold storage; (3) compress warm nodes into summaries and repoint edges; (4) emergency — evict non-critical nodes by ascending score, never critical-path nodes.
  5. Ancestry repair: when a pruned node is later requested, retrieve it from cold storage and re-attach its ancestors up to the hot boundary.

Suitability: excellent for coding agents (tool chains are explicit). Works in both million- and 200K-token windows; in smaller windows the compression phase fires more aggressively.

Algorithm B: Semantic clustering pruner

Purpose: group related turns into clusters, summarize clusters, and preserve the boundary messages that link clusters.

Logic:

  1. Segmentation: partition the conversation into fixed-size windows or natural boundaries.
  2. Embedding and clustering: embed each turn; cluster with HDBSCAN or k-means (k ≈ turns/10). Clusters represent episodes.
  3. Boundary detection: messages with edges crossing into another cluster are boundary messages and are kept verbatim as semantic bridges.
  4. Intra-cluster summarization: summarize each cluster (topic, decisions, files touched, tool calls, open questions) and replace non-boundary messages with the summary.
  5. Meta-clustering (optional): treat summaries as messages and re-cluster into themes, forming a pyramid.
  6. Retrieval on demand: when the current query is semantically distant from all summaries, expand the most relevant raw cluster from cold storage.

Suitability: best for open-ended research or brainstorming where tool chains are loose and topics shift. Less effective for tight coding loops where exact tool-parameter history matters.

Algorithm C: Hybrid proactive/reactive multi-tier pruner

Purpose: maintain explicit hot/warm/cold tiers with proactive triggers and a reactive emergency fallback.

Data structures:

class TierConfig:
    hot_max_tokens: int      # e.g., 200_000 for a 1M window
    warm_max_tokens: int     # e.g., 300_000
    cold_storage: str        # external store (SQLite, JSONL)
    trigger_yellow: float    # 0.70
    trigger_orange: float    # 0.85
    trigger_red: float       # 0.95

class MultiTierPruner:
    hot: List[ContextNode]
    warm: List[SummaryNode]
    cold: ColdStorage
    graph: DependencyGraph
    config: TierConfig

Logic:

  1. Token audit loop: after each turn, compute utilization = used / window.
  2. Proactive triggers: at yellow, run Algorithm B over the oldest half of hot context and move summaries to warm; at orange, run Algorithm A to detach tool outputs and compress file reads to symbolic references; force a yellow pass every ~15 turns regardless of utilization.
  3. Reactive triggers: at red (or on an API warning), score non-critical hot nodes and evict the lowest directly to cold, then trim warm summaries to the most recent and most query-relevant. If a pending multi-step operation would overflow, pause it, prune, then resume.
  4. Warm-tier refresh: when a cold item is retrieved, promote it to warm; re-archive the oldest warm summary if needed.
  5. Continuity markers: after any pruning event, prepend a short marker to the next message ("Context pruned: 3 episodes summarized, 2 files now referenced by hash. Active: auth refactor in login.py.").

Suitability: the recommended production architecture — it combines the structural safety of Algorithm A with the compression efficiency of Algorithm B. Proactive triggers keep the working set small enough that reactive events are rare. For 200K-token windows, lower the thresholds (yellow 60%, orange 75%, red 90%).

Evaluation and Benchmarking

Metrics for pruning quality

Metric Definition Measurement Method
Task completion rate Tasks resolved after pruning Agent task benchmarks
User-intent preservation Does the agent still obey pre-pruning constraints? Human eval or automated needle tests
Repeated file-read reduction Reduction in redundant file reads vs. baseline Instrument file I/O; compare tokens
Repeated tool-call reduction Reduction in redundant tool calls Instrument tool invocations
Token efficiency 1 − (tokens_with / tokens_without) API token counting
Compression ratio original / pruned Per-task measurement
AST correctness (code) Pruned code contexts that remain syntactically valid Parse after pruning
Interaction rounds Turns to completion Trajectory analysis

Synthetic benchmark designs

  • Needle-in-haystack agent benchmark: generate 100-turn sessions with ~20 constraints embedded at random turns; prune at turns 50 and 100; query each needle at turn 101; score recall.
  • Tool-chain resilience benchmark: construct multi-step tasks needing 5–10 sequential tool calls; inject a pruning event mid-chain; measure whether the agent completes without repeating steps.
  • Context-drift benchmark: pair related tasks; solve task 1, prune, solve task 2; measure whether the agent reuses relevant context from task 1.
  • Delta/diff efficacy benchmark: simulate reading 20 files then re-reading 10; compare full re-read versus diff references; target over 80% reduction for unchanged files.

Risks and Mitigations

Over-pruning and loss of nuance

Aggressive summarization drops critical modifiers (e.g., "use type hints everywhere" becomes "add type hints"). Token-level compression methods can dramatically reduce code AST correctness compared to structure-aware pruning. Mitigate by preserving verbatim-grounded artifacts (decisions, constraints, todos), using line-level or structural pruning for code, and allowing users to pin specific messages.

Breaking tool chains or reasoning paths

Pruning a tool result that a pending call depends on causes the agent to hallucinate or repeat the call. Dependency-graph tracking (Algorithm A) is non-negotiable for tool-augmented agents: before any eviction, compute the transitive closure of pending tool calls and protect all ancestors. Keep tool-result summaries that preserve key fields (return code, stdout snippet, error flag) even when full output is evicted.

Computational overhead of relevance scoring

Embedding every turn and computing centrality adds latency. Use lightweight embeddings (e.g., a small GTE model) for the pruning pipeline, score incrementally within a sliding window of the newest turn, and batch embedding calls. For inline (non-spawned) operation, keep overhead under ~500 ms per turn.

Trust degradation

Users lose confidence if they sense the agent "forgot." Emit transparency messages after pruning, maintain a user-visible memory ledger summarizing what is in each tier, and never prune user-stated goals without confirmation.

Building an Adaptive Context Pruner: A Prioritized Roadmap

Immediate: delta/diff file caching

Implement a file_cache layer storing SHA-256 hashes and compressed contents for every file read; on re-read return a symbolic reference or unified diff. Zero ML overhead, fastest token win (large reductions on file re-reads), directly applicable to file-read-heavy agent workflows. Low cost — a pure software layer with no external model calls.

Short-term: dependency-graph-based pruner

Build Algorithm A. Track every tool call, file read, and user query as DAG nodes; evict non-critical leaves by composite relevance; protect tool-chain ancestors. Highest impact on task completion and the structural backbone that makes all other pruning safe. Medium cost — requires a graph structure and centrality scoring; an embedding model is optional (keyword heuristics suffice for a first version).

Medium-term: hybrid multi-tier system

Combine Algorithms A and B into Algorithm C. Add proactive yellow/orange/red triggers based on token-count utilization; maintain hot/warm/cold tiers with explicit budgets and external cold storage. Proactive pruning prevents disruptive emergency events on large windows. Medium-to-high cost — threshold tuning and per-task weight calibration.

Deferred: task-aware neural pruner

Train or fine-tune a lightweight skimmer (sub-1B parameters) to score context lines or messages given a goal hint. Highest theoretical compression ratio, but it requires labeled training data and inference infrastructure. High cost — not recommended until the structural pruner is mature.

Sources