Virtual Memory for AI Agents in Production: Context Window Paging, Working Set Estimation, and Hierarchical Storage Architectures

As autonomous AI agents shift from single-turn chat interactions to long-horizon workflows spanning days, weeks, or millions of execution steps, managing context has become the primary operational bottleneck. While modern foundation models support nominal context windows ranging from 128k to over 1M tokens, treating the active context window as an unbounded append-only log degrades performance.

Empirical studies show that attention mechanisms suffer from severe positional degradation, commonly termed the "lost in the middle" effect (Liu et al., 2023), where retrieval accuracy drops significantly when relevant context is buried among extraneous conversational history. Furthermore, passing hundreds of thousands of accumulated prompt tokens on every agent step causes quadratic attention compute scaling in dense models, massive Key-Value (KV) cache allocation overheads in inference engines (Kwon et al., 2023), and prohibitive API inference costs.

To solve these constraints, production agent architectures are adopting classic operating system virtual memory paradigms. Pioneered by systems like MemGPT (Packer et al., 2023) and generalized in agent operating systems such as AIOS (Mei et al., 2024), virtual context management decouples an agent's unbounded addressable state from the fixed physical context window of the underlying model.

The Agent Memory Hierarchy

Traditional operating systems decouple fast, scarce physical RAM from abundant, slower block storage through hierarchical tiers. Agent virtual memory mirrors this architecture by partitioning state into three primary layers:

+---------------------------------------------------------------+
|             L1: Active Working Context (In-Prompt RAM)        |
|  - System Core Directive (Role, Rules, Safety Constraints)    |
|  - Working Scratchpad (Persona, Current Task, Pinned State)   |
|  - Active FIFO Turn Buffer (Last K Interaction Messages)      |
+---------------------------------------------------------------+
                                |
               Page In / Out    |  Demand Paging & Eviction
                                v
+---------------------------------------------------------------+
|         L2: Virtual Working Set (Local Fast Storage)          |
|  - High-Speed Key-Value Cache / Session State Engine          |
|  - Embedded FTS5 SQLite Store / Local Vector Graph            |
|  - Episodic Trajectory Chunks & Active Tool Schemas           |
+---------------------------------------------------------------+
                                |
               Async Flush      |  Hierarchical Archival
                                v
+---------------------------------------------------------------+
|         L3: Archival Cold Storage (Persistent Stores)         |
|  - Enterprise Knowledge Base / Vector Database (pgvector/Qdrant)|
|  - Multi-Modal Asset Store / Object Storage (S3 / Blob)       |
|  - Long-Term Historical Logs & Checkpoints                    |
+---------------------------------------------------------------+

1. L1: Active Working Context (Physical Prompt Buffer)

The L1 tier represents the physical context window passed directly into the model forward pass. It is strictly bounded (typically 4,000 to 16,000 tokens) to ensure rapid time-to-first-token (TTFT), preserve high attention concentration, and maximize prefix-caching hit rates on inference gateways. It comprises:

• System Core Directive: Immutable operating instructions and tool calling declarations.
• Working Scratchpad (Core Memory): Structured key-value text slots representing identity, user facts, and ongoing task milestones that the model can rewrite in place.
• FIFO Turn Queue: A sliding window of the most recent message turns and immediate tool execution returns.

2. L2: Virtual Working Set (Fast Local Storage)

The L2 tier resides out-of-context in process memory or high-speed local disk (such as embedded SQLite with FTS5 or an in-memory vector index). It holds recently evicted conversation blocks, intermediate tool outputs, and structured state machines that are not needed on every turn but are likely to be referenced within the same task execution horizon.

3. L3: Persistent Archival Storage (Cold Tier)

The L3 tier comprises durable external databases, such as vector stores (Pinecone, Qdrant, pgvector) and document stores. It contains unbounded historical memory, long-term domain knowledge bases, and multi-session historical transcripts. Access latencies are higher (50ms to 200ms), but storage capacity is virtually limitless.

Working Set Estimation in Agent Workflows

In classical computer systems, Peter Denning's Working Set Model (Denning, 1968) defines the working set $W(t, \tau)$ as the collection of memory pages referenced by a process during the time interval $(t-\tau, t)$. In agentic execution, an analogous working set represents the subset of episodic memory, variable bindings, and reference documents required to resolve the current reasoning step without triggering context starvation or attention distraction.

                  Temporal Locality Window (tau)
           t-tau -----------------------------> t (Current Step)
                 [Page A]  [Page C]  [Page A]  [Page B]
                                |
                                v
           Working Set W(t, tau) = {Page A, Page B, Page C}
           (Loaded into L1 Working Context; all others paged to L2)

Working set estimation for agents combines two orthogonal dimensions:

1. Temporal Locality: Context items referenced within recent execution steps have a high probability of being referenced again. This is tracked via exponential decay access counters:

$$S_{temporal}(p, t) = \sum_{t_i \in \text{Accesses}(p)} \exp\left(-\lambda (t - t_i)\right)$$ where $p$ is the context page, $t_i$ is the timestep of access, and $\lambda$ is the decay constant.

1. Semantic Locality: Memory blocks possessing high vector cosine similarity or hybrid BM25 lexical relevance to the active scratchpad state are dynamically predicted as part of the imminent working set:

$$S_{semantic}(p, q) = \alpha \cdot \text{Sim}_{dense}(e_p, e_q) + (1 - \alpha) \cdot \text{Score}_{BM25}(p, q)$$ where $q$ is the query formulated from the active working scratchpad and $e_p, e_q$ are their respective dense embeddings.

When the combined locality score drops below an admission threshold, the page is flagged for eviction to L2 storage.

Demand Paging and Page Fault Handling

When an agent requires information not present in its L1 prompt buffer, a context page fault occurs. Production systems implement page fault resolution through two complementary mechanisms:

[Agent Execution Step]
         |
         |---> Scenario A: Agent requests missing historical fact / tool output
         |     (Explicit Tool-Driven Page Fault: archival_search / conversation_page)
         |
         |---> Scenario B: In-flight token usage crosses high-water mark (e.g., 85% of L1 budget)
               (Implicit System Interrupt: Trigger eviction pipeline & flush oldest FIFO blocks)

Explicit Function-Driven Paging

As demonstrated in MemGPT (Packer et al., 2023), the LLM acts as the operating system kernel and manages its own context by invoking paging primitives. These include:

• core_memory_append(key, value) / core_memory_replace(key, value): Updates the in-prompt scratchpad.
• archival_memory_search(query, page, limit): Queries L3 storage and returns matching snippets into the next turn's scratch buffer.
• conversation_history_search(query, start_date, end_date): Pages historical dialogue frames from L2/L3 into the active working window.

Implicit System Interrupts

When the active context window approaches its predefined token budget (e.g., 85% of allocated L1 capacity), the agent middleware raises an implicit interrupt before executing the next model forward pass. The runtime executes an automated page replacement routine, serializes the oldest FIFO messages into L2 storage, and injects a compact paging summary into the prompt.

Page Replacement and Eviction Policies

Selecting which pages to evict from L1 is critical. Naive FIFO eviction often discards early constraints or vital variable definitions, whereas pure LRU can thrash if long-running loops periodically reference initialization parameters.

+-------------------+---------------------------------------------------------+
| Policy            | Mechanism & Trade-offs                                  |
+-------------------+---------------------------------------------------------+
| Paged-LRU         | Evicts the least recently attended memory block.        |
|                   | High efficiency; susceptible to bursty scan pollution.  |
+-------------------+---------------------------------------------------------+
| Semantic CLOCK    | Uses a circular buffer with reference bits and semantic  |
|                   | utility scores. Avoids evicting high-relevance blocks.  |
+-------------------+---------------------------------------------------------+
| Hierarchical      | Rather than raw eviction, summarizes blocks into 10%    |
| Compaction        | length meta-tokens before pushing to L2.                |
+-------------------+---------------------------------------------------------+

The Semantic CLOCK Algorithm

To avoid expensive global sorting of all prompt blocks on every turn, production runtimes utilize a modified CLOCK replacement algorithm:

1. Context blocks in the turn buffer are arranged in a circular list with a usage bit ($u \in \{0, 1\}$) and a priority flag ($P \in \{\text{Low, Normal, Pinned}\}$).
2. When space is required, the clock pointer scans the buffer:

• If a block has $P = \text{Pinned}$ (such as system rules or active plan invariants), it is skipped.
• If $u = 1$, the pointer clears the bit ($u \leftarrow 0$) and advances.
• If $u = 0$, the block is selected for eviction, written to L2 storage, and replaced with an empty slot.

Recursive Memory Compaction

When context blocks are evicted, simply dropping raw text loses relational continuity. Modern agent runtimes employ hierarchical recursive summarization (Snell et al., 2024). The evicted chunk is passed to an asynchronous background worker that compresses the text into structured semantic assertions, updating the L2 entity graph before freeing L1 tokens.

Write-Back Policies and Memory Coherence

In multi-agent systems and multi-threaded agent workflows, memory mutations present concurrency challenges. If an agent updates its working scratchpad during a speculative tool execution branch that subsequently fails, stale or incorrect data must not pollute persistent storage.

                      +-----------------------------+
                      | Agent Modifies State Record |
                      +-----------------------------+
                                     |
                    +----------------+----------------+
                    |                                 |
                    v                                 v
        [Write-Through Mode]                 [Write-Back Mode]
  - Synchronous write to L2/L3         - Marked as "Dirty Page" in L1
  - High latency (100-250ms)           - Zero execution overhead in-turn
  - Strict consistency                 - Flushed on commit or task end
  - Rollback on execution abort        - Discarded on branch cancellation

1. Write-Through Caching

Every update to core memory or the agent scratchpad is synchronously committed to the underlying L2/L3 datastore before the next LLM turn proceeds. This guarantees durability and cross-agent consistency but introduces a 50ms to 200ms latency penalty per memory mutation.

2. Write-Back Caching with Dirty Page Tracking

Memory edits remain local to the L1 scratchpad and are marked with a dirty flag. The runtime flushes dirty pages to L2/L3 only when:

• A task step successfully completes verification checks.
• A context page fault forces an eviction of that specific block.
• A periodic checkpoint boundary is reached.

If an execution branch fails or triggers a tool exception, the runtime executes a transactional rollback, discarding dirty in-prompt modifications and restoring the last clean checkpoint from L2.

Production Implementation Reference

The following Python architecture demonstrates a modular virtual memory manager for long-running agents, integrating an L1 working context buffer, SQLite-backed L2 virtual storage, and an explicit paging interface.

import sqlite3
import time
from typing import List, Dict, Any, Optional

class AgentVirtualMemoryManager:
    """
    Manages hierarchical agent virtual memory across L1 working context
    and L2 disk-backed persistent virtual working set.
    """
    def __init__(self, db_path: str = ":memory:", l1_token_limit: int = 8000):
        self.l1_token_limit = l1_token_limit
        self.core_memory: Dict[str, str] = {}
        self.active_turn_buffer: List[Dict[str, Any]] = []
        
        # Initialize L2 SQLite database with Full-Text Search (FTS5)
        self.conn = sqlite3.connect(db_path)
        self._init_l2_storage()

    def _init_l2_storage(self) -> None:
        with self.conn:
            self.conn.execute("""
                CREATE TABLE IF NOT EXISTS l2_pages (
                    page_id TEXT PRIMARY KEY,
                    section TEXT,
                    content TEXT,
                    access_count INTEGER DEFAULT 1,
                    last_accessed REAL,
                    dirty INTEGER DEFAULT 0
                )
            """)
            self.conn.execute("""
                CREATE VIRTUAL TABLE IF NOT EXISTS l2_fts USING fts5(
                    page_id UNINDEXED,
                    content,
                    tokenize='porter unicode61'
                )
            """)

    def set_core_memory(self, key: str, value: str) -> None:
        """Updates L1 core scratchpad memory slot."""
        self.core_memory[key] = value

    def append_turn(self, role: str, content: str, tokens: int) -> None:
        """Appends a new turn to L1 buffer and triggers page replacement if over limit."""
        turn_entry = {
            "role": role,
            "content": content,
            "tokens": tokens,
            "timestamp": time.time(),
            "referenced": True
        }
        self.active_turn_buffer.append(turn_entry)
        self._enforce_working_set_budget()

    def _enforce_working_set_budget(self) -> None:
        """Evicts oldest turns to L2 when L1 token consumption exceeds budget."""
        current_tokens = sum(turn["tokens"] for turn in self.active_turn_buffer)
        
        while current_tokens > self.l1_token_limit and len(self.active_turn_buffer) > 2:
            # Evict oldest turn (preserving the most recent active instructions)
            evicted = self.active_turn_buffer.pop(0)
            page_id = f"turn_{int(evicted['timestamp'] * 1000)}"
            
            with self.conn:
                self.conn.execute("""
                    INSERT OR REPLACE INTO l2_pages (page_id, section, content, last_accessed, dirty)
                    VALUES (?, 'conversation', ?, ?, 0)
                """, (page_id, evicted["content"], time.time()))
                
                self.conn.execute("""
                    INSERT INTO l2_fts (page_id, content) VALUES (?, ?)
                """, (page_id, evicted["content"]))
                
            current_tokens -= evicted["tokens"]

    def demand_page_search(self, query: str, limit: int = 3) -> List[Dict[str, Any]]:
        """Handles explicit page faults by searching L2 virtual storage via FTS5."""
        cursor = self.conn.cursor()
        cursor.execute("""
            SELECT p.page_id, p.content, p.last_accessed
            FROM l2_fts f
            JOIN l2_pages p ON f.page_id = p.page_id
            WHERE l2_fts MATCH ?
            ORDER BY rank
            LIMIT ?
        """, (query, limit))
        
        results = []
        now = time.time()
        for row in cursor.fetchall():
            page_id, content, _ = row
            # Update access statistics
            cursor.execute("""
                UPDATE l2_pages 
                SET access_count = access_count + 1, last_accessed = ? 
                WHERE page_id = ?
            """, (now, page_id))
            results.append({"page_id": page_id, "content": content})
            
        self.conn.commit()
        return results

    def assemble_prompt(self, system_prompt: str) -> str:
        """Compiles the active L1 physical prompt buffer for the LLM forward pass."""
        core_block = "\n".join(f"<{k}>\n{v}\n</{k}>" for k, v in self.core_memory.items())
        turns_block = "\n".join(f"{t['role'].upper()}: {t['content']}" for t in self.active_turn_buffer)
        
        return (
            f"=== SYSTEM DIRECTIVE ===\n{system_prompt}\n\n"
            f"=== CORE MEMORY (WORKING SET) ===\n{core_block}\n\n"
            f"=== ACTIVE CONVERSATION ===\n{turns_block}\n"
        )

Architectural Trade-offs and Best Practices

Deploying virtual context management in enterprise agent swarms requires balancing token economics, execution latency, and state coherence.

+---------------------------+-----------------------+------------------------+
| Metric / Dimension        | Unbounded Linear Log  | Paged Virtual Memory   |
+---------------------------+-----------------------+------------------------+
| Prompt Token Growth       | $O(N)$ Linear Accum.  | $O(1)$ Bounded Buffer  |
| Attention Focus           | Degrades over depth   | High concentration     |
| Average Cost per Step     | Scales with history   | Flat, predictable      |
| Fault-Handling Overhead   | 0ms (all in context)  | 50-150ms on page fault |
| Implementation Complexity | Trivial               | High (Requires VM/FTS) |
+---------------------------+-----------------------+------------------------+

Key Engineering Rules for Production Deployments:

1. Budget L1 for Maximum Prefix Reuse: Align the static portions of the L1 working context (system directives, schema definitions) so they remain byte-for-byte identical across turns, ensuring 90%+ prompt cache hits on providers like Anthropic and OpenAI.
2. Cap L1 FIFO Window at 10-15 Turns: Beyond 15 interaction turns, raw conversation history produces diminishing retrieval utility while doubling prompt latency. Push older context to L2 immediately.
3. Enforce Strict Schema Typing on Scratchpads: Unstructured scratchpad text inevitably drifts into inconsistent formatting. Structure core memory keys using strict JSON Schema or Pydantic representations.
4. Instrument Page Fault Rates: Track the ratio of demand paging queries to overall agent turns. A page fault rate exceeding 30% indicates an undersized L1 token budget, causing thrashing between L1 prompt assembly and L2 storage queries.

Sources

• Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., & Gonzalez, J. E. (2023). MemGPT: Towards LLMs as Operating Systems. arXiv preprint arXiv:2310.08560.
• Mei, K., Li, Z., Xu, S., Ye, R., Ge, Y., & Zhang, Y. (2024). AIOS: LLM Agent Operating System. arXiv preprint arXiv:2403.16971.
• Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP '23, arXiv:2309.06180.
• Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, arXiv:2307.03172.
• Denning, P. J. (1968). The Working Set Model for Program Behavior. Communications of the ACM, 11(5), 323-333, doi:10.1145/363095.363141.
• Snell, C., Lee, J., Xu, K., & Kumar, A. (2024). Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. arXiv preprint arXiv:2408.03314.