Ephemeral File Systems for AI Coding Agents: Git Worktrees, Rootless OverlayFS, and Copy-on-Write Isolation

Autonomous AI coding agents frequently execute arbitrary shell commands, modify source code, install third-party dependencies, and run test suites. Granting an unconstrained agent direct write access to a developer's active working tree creates immediate operational hazards: accidental destruction of untracked files, workspace corruption from speculative refactoring, and state leaks across parallel tasks.

Heavyweight virtualization solutions like full virtual machines or freshly initialized container images solve the containment problem but introduce severe latency penalties. Spinning up a cold container or microVM can take several seconds to minutes when factoring in dependency installations, which degrades the interactive throughput required for multi-step agent loops.

Production agent architectures solve this latency-isolation trade-off through ephemeral filesystem virtualization. By combining Git worktrees, rootless OverlayFS mounts, and Linux kernel Landlock LSM rules, systems can provision isolated, sub-100-millisecond execution sandboxes with zero-cost rollbacks.

Architecture 1: Git Worktrees with Shared Object Stores

The simplest mechanism for isolating parallel coding tasks is the Git worktree. Traditional git workflows require cloning an entire repository or sequentially switching branches within a single working directory. A git worktree allows multiple working trees to be attached to the same repository database.

# Provision an isolated worktree for agent task execution
git worktree add -b agent/task-482 /tmp/worktrees/task-482 origin/main

When an agent executes inside a dedicated worktree:

Shared Object Database: All worktrees share the primary .git/objects directory. Commits, blobs, and trees created by one agent are immediately available to the local repository without network cloning or duplicated disk storage.
Independent Index and Head: Each worktree maintains its own .git/worktrees/<id>/index and HEAD file. An agent can modify files, stage changes, and commit branches in parallel without altering other active workspaces.
Sub-50ms Allocation: Worktree creation requires only writing a lightweight index and checking out target files from the local object database, completing in tens of milliseconds.

The Dependency Isolation Bottleneck

While Git worktrees isolate version-controlled source files, they do not isolate untracked dependencies such as node_modules, Python virtual environments (.venv), or build caches (target/, build/). Running package installations inside every ephemeral worktree eliminates the speed advantage of worktrees.

Engineering teams address dependency overhead through three patterns:

Copy-on-Write Linking: On filesystems supporting copy-on-write (such as Apple APFS via cp -c or Linux Btrfs and XFS via cp --reflink=always), dependency directories can be cloned into the worktree instantaneously. Storage blocks are shared until an agent modifies a dependency file.
Read-Only Symlink Trees: When lockfiles are guaranteed immutable for a task, the agent workspace symlinks the main repository's dependency folder directly.
Content-Addressable Package Caches: Tools like pnpm and uv maintain global, immutable package caches on disk. Ephemeral workspaces create hardlinks to this shared cache rather than downloading or duplicating packages.

Architecture comparison of Git worktrees and layered OverlayFS

Architecture 2: Layered Copy-on-Write with Rootless OverlayFS

When an agent requires total filesystem isolation, including the ability to delete or modify arbitrary system files without persisting changes, layered filesystems provide a cleaner boundary than Git worktrees alone.

OverlayFS is a union filesystem service in the Linux kernel that combines multiple directory trees into a single unified mount point.

Layer Anatomy

An agent OverlayFS setup consists of four distinct directories:

lowerdir (Read-Only Base): The original repository directory. The agent can read all project files, but the kernel enforces strict read-only access on the underlying storage.
upperdir (Writable Delta): An ephemeral directory on a fast scratch disk (such as tmpfs or an NVMe partition). Every file creation, modification, or write operation performed by the agent lands exclusively in this layer.
workdir (Atomic Operations): An empty scratch directory on the same filesystem as upperdir, required by the kernel to prepare files before switching them atomically into upperdir.
merged (Unified Target): The virtual directory presented to the agent.

# Mounting an ephemeral OverlayFS layer
mount -t overlay overlay \
  -o lowerdir=/srv/repo/main,upperdir=/tmp/agent-482/upper,workdir=/tmp/agent-482/work \
  /tmp/agent-482/merged

Unprivileged User Namespaces

Modern agent execution environments run daemonless and without root privileges. By combining unprivileged user namespaces (CLONE_NEWUSER and CLONE_NEWNS via unshare), an unprivileged agent process can map its own user ID to root inside the namespace and perform OverlayFS mounts without host root permissions.

# Create an unprivileged user + mount namespace and mount OverlayFS
unshare -U -m -r sh -c "
  mount -t overlay overlay -o lowerdir=/srv/repo,upperdir=/tmp/upper,workdir=/tmp/work /tmp/merged
  cd /tmp/merged && exec /bin/bash
"

Deletions via Whiteouts and the Rename Penalty

OverlayFS manages deletions in the read-only lowerdir by creating character devices with major/minor number 0,0, known as whiteout files (mknod c 0 0). When the agent executes rm src/app.py, the underlying lower file is untouched; the kernel places a whiteout marker in upperdir that hides the file from the merged view.

However, standard OverlayFS introduces an architectural constraint during directory renames. When an agent renames a directory that exists in lowerdir, the kernel cannot simply rename the directory metadata across layers. Unless the kernel option redirect_dir=on is enabled, the operation returns an EXDEV error ("Invalid cross-device link"). If enabled, or if handled in fallback mode, the kernel executes a recursive directory copy-up, copying every child file into upperdir. For large codebases, directory refactoring by an agent can trigger significant I/O spikes.

Architecture 3: Kernel-Level Restrictions with Landlock LSM and Seccomp

Filesystem mounts alone do not prevent an agent from traversing the host directory structure (for example, reading /etc/passwd, scanning /proc, or inspecting sensitive user configuration in ~/.ssh and ~/.aws).

Landlock is an unprivileged Linux Security Module (LSM) introduced in Linux 5.13 that enables processes to restrict their own filesystem access rights. Unlike traditional container access control, Landlock is enforced directly by the kernel at the system call layer without requiring root privileges.

Enforcing Sandboxed Paths

Before invoking an agent execution subprocess, the host supervisor configures a Landlock ruleset using three primary system calls:

landlock_create_ruleset() defines the scope of monitored actions (LANDLOCK_ACCESS_FS_READ, LANDLOCK_ACCESS_FS_WRITE, LANDLOCK_ACCESS_FS_EXECUTE).
landlock_add_rule() attaches rules to specific file descriptors using LANDLOCK_RULE_PATH_BENEATH.
landlock_restrict_self() seals the ruleset onto the current process and all future child processes.

// Conceptual Landlock enforcement in C
struct landlock_ruleset_attr attr = {
    .handled_access_fs = LANDLOCK_ACCESS_FS_READ | LANDLOCK_ACCESS_FS_WRITE | LANDLOCK_ACCESS_FS_EXECUTE,
};
int ruleset_fd = landlock_create_ruleset(&attr, sizeof(attr), 0);

// Grant full read/write/exec strictly to the agent workspace
struct landlock_path_beneath_attr path_attr = {
    .allowed_access = LANDLOCK_ACCESS_FS_READ | LANDLOCK_ACCESS_FS_WRITE | LANDLOCK_ACCESS_FS_EXECUTE,
    .parent_fd = open("/tmp/agent-482/merged", O_PATH | O_DIRECTORY),
};
landlock_add_rule(ruleset_fd, LANDLOCK_RULE_PATH_BENEATH, &path_attr, 0);

// Restrict system binaries to read-only execution
struct landlock_path_beneath_attr sys_attr = {
    .allowed_access = LANDLOCK_ACCESS_FS_READ | LANDLOCK_ACCESS_FS_EXECUTE,
    .parent_fd = open("/usr", O_PATH | O_DIRECTORY),
};
landlock_add_rule(ruleset_fd, LANDLOCK_RULE_PATH_BENEATH, &sys_attr, 0);

// Lock process permanently
landlock_restrict_self(ruleset_fd, 0);

Once landlock_restrict_self() executes, the restriction is irreversible. Even if an agent manages to compromise a child process or gain fake root inside a user namespace, the kernel drops any filesystem access outside /tmp/agent-482/merged and /usr. Pairing Landlock with seccomp prevents unauthorized system operations like ptrace injection or network interface manipulation.

State Checkpointing, Verification, and Rollback

The primary benefit of ephemeral filesystems in AI coding workflows is deterministic recovery from agent errors.

Agent Proposes Solution
        │
        ▼
Mount Ephemeral Overlay / Worktree
        │
        ▼
Agent Modifies Files & Runs Tests
        │
   ┌────┴────┐
   ▼         ▼
Tests Fail  Tests Pass
   │         │
   │         ▼
   │    Inspect Diff & Commit
   │    Export Patch to Base Repo
   │         │
   └────┬────┘
        │
        ▼
Unmount & Discard Ephemeral Layer
(Zero host residue)

Pre-Execution Baseline: An ephemeral worktree or OverlayFS mount is created at a known commit hash.
Speculative Execution: The agent implements code changes, updates configuration files, and executes verification commands (pytest, npm test, cargo check).
Verification Oracle:

Failure Path: If tests fail, linters report syntax breakage, or the agent enters a circular error state, the system unmounts the OverlayFS or removes the worktree (git worktree remove --force). All file mutations disappear immediately with zero leftover residue.
Success Path: If tests pass, the system extracts the delta directly from upperdir (or generates a patch via git diff), verifies the changes against security policies, and applies a clean atomic commit to the target branch.

Filesystem Isolation Comparison

Git Worktree: Creation latency is 20 to 50 milliseconds. Storage overhead is limited to the modified delta plus checked-out working files. Isolation relies on standard Unix user permissions. Rollback is performed via git worktree remove. Dependency sharing relies on symlinks or copy-on-write clones. Supported on any operating system with Git installed.
Rootless OverlayFS: Creation latency is 5 to 15 milliseconds. Storage overhead is restricted to modified delta files in upperdir. Isolation is enforced via unprivileged user namespaces and virtual filesystem layers. Rollback requires only deleting upperdir. Dependencies are shared through the read-only lowerdir. Requires Linux 5.11+ unprivileged OverlayFS support.
Btrfs / XFS Reflink: Creation latency is 10 to 30 milliseconds. Storage overhead is block-level delta only. Isolation is maintained at the filesystem block level. Rollback is executed by deleting subvolume snapshots. Dependencies share underlying physical extents. Requires Linux Btrfs or XFS filesystems.
MicroVM (Firecracker): Creation latency is 150 to 500 milliseconds. Storage overhead is a full root filesystem disk image or sparse backing file. Isolation is enforced by hardware virtualization through Linux KVM. Rollback is executed by tearing down the virtual machine instance. Dependencies must be baked into the container image layers. Requires Linux KVM support.

Production Engineering Recommendations

Use Git Worktrees for Parallel Branching: When agents collaborate on separate features that will ultimately become pull requests, Git worktrees provide native Git integration without requiring filesystem mounting privileges.
Use Rootless OverlayFS for High-Frequency Loops: For benchmark harnesses (such as SWE-bench evaluation runners) and speculative exploration loops where hundreds of short-lived tasks run every hour, OverlayFS on tmpfs delivers near-zero creation overhead and instantaneous cleanup.
Always Bind Sandboxes with Landlock: Never rely solely on path validation inside agent code. Enforcing Landlock at the process boundary guarantees that rogue bash commands cannot access parent directories or host secrets.
Decouple Package Caches from Task Worktrees: Configure package managers (pnpm store, uv cache, cargo cache) to point to a centralized, read-only mounted directory to keep ephemeral worktree creation under 50 milliseconds.