Tool-Call Caching in Production AI Agents: Architecture, State Invalidation, and Latency Economics

In multi-turn agentic architectures such as ReAct, plan-and-solve swarms, and autonomous coding runtimes, large language models spend significant time waiting on external tool execution. While prompt caching and prefix KV-cache reuse have reduced inference costs for repeated prompt contexts, they do not optimize the downstream execution layer. When an agent queries a database, scrapes a webpage, executes a sandboxed bash command, or retrieves embeddings, external tool latency frequently accounts for 70% to 90% of total wall-clock runtime.

Unchecked tool calling introduces dual overheads: cumulative network and compute latency that degrades interactive response times, and compounding API billing from external service providers. Implementing tool-call caching and execution memoization allows agent frameworks to bypass redundant external invocations. However, applying classic web caching directly to LLM tool execution introduces complex failure modes, including state desynchronization, stale read propagation, and dangerous memoization of mutative operations.

Taxonomy of Tool Invocations

A production caching layer must classify tool calls by their execution semantics and side effects. Treating all tool invocations identically risks either cache pollution or data corruption.

• Informational Tools (Idempotent and Read-Only): Invocations such as database SELECT queries, file reads, knowledge base lookups, static API queries, and deterministic math operations produce no external side effects. Given identical input parameters and stable underlying state, these calls can be safely cached with structured time-to-live (TTL) policies or tag-based invalidation.
• Command Tools (Mutative and State-Changing): Operations such as file writes, database updates, payment processing, sending emails, or triggering deployments modify system state. These operations must never be served from a cached result. Instead, command tools require transactional execution with idempotency keys and must act as invalidation triggers for dependent read caches.
• Semi-Deterministic and Ephemeral Tools: Live web searches, price tickers, and dynamic web scrapers exhibit varying freshness requirements. Caching these calls requires aggressive TTL decay or adaptive validation headers (such as ETag or Last-Modified checks) to prevent hallucinated agent reasoning based on outdated information.

Recent formalizations in protocols like the Model Context Protocol (MCP) distinguish between informational read resources and tool commands, providing a structured basis for cache gating at the protocol level.

Canonicalization and Cache Key Construction

In naive implementations, developers often hash the raw JSON payload produced by the LLM to generate a cache key. This approach causes low cache hit ratios due to the non-deterministic nature of model token generation.

# Naive hashing fails due to arbitrary key ordering and formatting
{"symbol": "NVDA", "interval": "1d", "limit": 100}
{"limit": 100, "symbol": "NVDA", "interval": "1d"}

Even when requesting structured outputs, LLMs frequently vary JSON key ordering, insert discretionary whitespace, or emit varying numeric formats (such as 100 versus 100.0). A production cache key pipeline must apply strict canonicalization:

1. Schema-Guided Key Sorting: Parse the invocation arguments into an intermediate schema structure and serialize with alphabetically sorted keys.
2. Type Coercion and Normalization: Enforce strict type conversions (integers, normalized floats, lowercased strings where case-insensitive).
3. Context Stripping: Remove ephemeral session tokens, conversation trace IDs, and client-generated timestamps from the argument payload before hashing.
4. Hashing: Generate a deterministic digest (such as SHA-256) over the tuple of (tool_name, tool_version, canonical_arguments).

For fuzzy queries, such as vector search or natural language retrieval, exact hash matching can be paired with an optional semantic layer. By embedding query strings and applying cosine similarity thresholds (typically 0.95 or higher), agents can identify semantically identical read queries. However, semantic argument matching adds 10ms to 30ms of embedding latency and risks false-positive cache hits, making it best suited for high-latency external search APIs.

Adaptive Admission and Value-Aware Eviction

Traditional cache replacement policies such as Least Recently Used (LRU) and Least Frequently Used (LFU) assume that all cache misses carry uniform cost. In tool execution, costs are wildly heterogeneous. A local AST parse may take 2 milliseconds and cost $0.00, whereas an enterprise web search or multi-hop data extraction API may take 2,500 milliseconds and cost $0.01 per call.

Evicting a slow, expensive API result to retain a lightweight local calculation degrades system efficiency. To resolve this, researchers have proposed adaptive frameworks such as ToolCaching (arXiv:2601.15335). ToolCaching implements the Value-Aware Admission and Eviction (VAAC) algorithm, which models cache admission as a Multi-Armed Bandit problem using the UCB1 algorithm to evaluate request cacheability dynamically.

# Conceptual value-aware eviction scoring
def calculate_eviction_score(entry, current_time):
    # Higher score = higher retention value
    latency_weight = entry.execution_time_ms
    cost_weight = entry.api_dollar_cost * 1000  # Scale dollar cost
    frequency = entry.hit_count
    age_seconds = max(1, current_time - entry.last_accessed_timestamp)
    payload_size_kb = max(1, entry.response_size_bytes / 1024)
    
    caching_value = (latency_weight + cost_weight) * frequency
    eviction_priority = caching_value / (payload_size_kb * (age_seconds ** 0.5))
    return eviction_priority

Under value-aware eviction, entries are ranked not merely by recency, but by the cumulative execution time and dollar expenditure saved per unit of storage memory. Empirical evaluations in research benchmarks show that value-aware admission and eviction can improve effective cache hit ratios by up to 11% and decrease end-to-end agent latency by up to 34% compared to standard LRU baselines.

Dependency Invalidation and Multi-Agent Swarms

The primary risk of aggressive tool-call memoization in autonomous agents is state desynchronization. In multi-step workflows, an agent frequently reads state, executes reasoning, and modifies state. If an agent executes write_file("config.py"), subsequent calls to read_file("config.py") must not return the cached pre-modification content.

To maintain consistency without disabling caching entirely, production agent frameworks implement graph-based dependency tracking and tag invalidation, as detailed in hierarchical caching architectures for agentic workflows.

   [Tool: search_code("auth")] ---> Cached (Tags: repo:backend, path:auth/)
   [Tool: read_file("auth/jwt.py")] ---> Cached (Tags: repo:backend, path:auth/jwt.py)
                   │
                   ▼
   [Tool: write_file("auth/jwt.py", content)] (Mutative Command)
                   │
                   ▼
   [Invalidation Event Bus: PURGE tags "path:auth/jwt.py", "repo:backend"]
                   │
                   ▼
   Subsequent reads forced to execute against fresh disk state

Invalidation Mechanics

1. Tagging on Write-Read Overlap: Informational tool calls register cache tags corresponding to the entity or scope they access (such as user:1042, repo:backend:auth/jwt.py, or table:orders).
2. Mutative Invalidation Hooks: Command tools publish invalidation events across a Redis/Valkey pub/sub bus upon successful completion, purging matching tags across all worker nodes.
3. Cross-Agent Shared Memoization: In multi-agent swarms (e.g., a planner delegating tasks to multiple parallel subagents), a centralized cache allows subagents to reuse expensive research or analysis artifacts without duplicating API invocations.

Production Implementation Blueprint

A robust tool-caching layer is typically implemented as interceptor middleware wrapping tool execution interfaces.

import hashlib
import json
import time
from typing import Any, Callable, Dict, Optional
import redis

class ToolCacheManager:
    def __init__(self, redis_client: redis.Redis, default_ttl: int = 3600):
        self.redis = redis_client
        self.default_ttl = default_ttl

    def _canonicalize_args(self, args: Dict[str, Any]) -> str:
        # Sort keys and ensure deterministic JSON formatting
        return json.dumps(args, sort_keys=True, separators=(",", ":"))

    def _generate_key(self, tool_name: str, args: Dict[str, Any]) -> str:
        canonical_str = self._canonicalize_args(args)
        arg_hash = hashlib.sha256(canonical_str.encode("utf-8")).hexdigest()
        return f"tool_cache:{tool_name}:{arg_hash}"

    def execute_with_cache(
        self,
        tool_name: str,
        tool_fn: Callable,
        args: Dict[str, Any],
        is_mutative: bool = False,
        invalidation_tags: Optional[list[str]] = None,
        custom_ttl: Optional[int] = None,
    ) -> Any:
        if is_mutative:
            # Execute command directly and invalidate dependent tags
            result = tool_fn(**args)
            if invalidation_tags:
                for tag in invalidation_tags:
                    self._invalidate_tag(tag)
            return result

        cache_key = self._generate_key(tool_name, args)
        cached_entry = self.redis.get(cache_key)

        if cached_entry:
            payload = json.loads(cached_entry)
            return payload["result"]

        start_time = time.perf_counter()
        result = tool_fn(**args)
        execution_duration = time.perf_counter() - start_time

        # Store result with execution metadata
        cache_data = {
            "result": result,
            "created_at": time.time(),
            "execution_time_ms": round(execution_duration * 1000, 2),
        }
        ttl = custom_ttl or self.default_ttl
        self.redis.setex(cache_key, ttl, json.dumps(cache_data))
        return result

    def _invalidate_tag(self, tag: str) -> None:
        # Invalidate all keys mapped to this tag
        tag_key = f"cache_tag:{tag}"
        members = self.redis.smembers(tag_key)
        if members:
            self.redis.delete(*members)
            self.redis.delete(tag_key)

Production Observability and Failure Budgets

Deploying tool-level caching in production requires specialized telemetry to avoid silent failures:

• Cache Hit and Bypassed Latency: Track both raw hit ratio and aggregate wall-clock milliseconds saved. A 15% hit ratio on 2,000ms search tools provides greater user-facing benefit than an 80% hit ratio on 5ms local calculations.
• Cost Savings Tracking: Attribute dollar savings based on upstream API price tables to quantify return on infrastructure investment.
• Stale Hit Anomaly Detection: Monitor downstream agent retry rates. If an agent repeatedly calls alternative tools or fails validation steps immediately after receiving a cached tool response, automated circuit breakers should invalidate the associated cache namespace.

Sources

• ToolCaching: Towards Efficient Caching for LLM Tool-calling (arXiv:2601.15335)
• Hierarchical Caching for Agentic Workflows: A Multi-Level Architecture to Reduce Tool Execution Overhead (MDPI)
• ReAct: Synergizing Reasoning and Acting in Language Models (arXiv:2210.03629)
• Model Context Protocol Specification (Anthropic)