Confidential LLM Inference in Production: Hardware TEEs, GPU Enclaves, Attestation, and Serving Performance Trade-Offs

Deploying large language models in multi-tenant cloud environments introduces a fundamental security boundary problem. Standard transport encryption (TLS) secures prompts in transit, and encryption-at-rest protects checkpoints on disk, but model weights, prompt tokens, and key-value (KV) caches exist in plaintext within system memory during active inference. For organizations processing protected health information (PHI), proprietary intellectual property, or regulated financial records, host operating system compromise or untrusted hypervisor access represents an unacceptable attack surface.

Confidential computing addresses this vulnerability through hardware-enforced Trusted Execution Environments (TEEs). While CPU-level TEEs like AMD SEV-SNP and Intel TDX have matured for general enterprise workloads, confidential LLM serving requires extending cryptographic isolation across the PCIe bus and into high-bandwidth GPU memory. Implementing end-to-end confidential inference introduces distinct architectural requirements, composite attestation workflows, and measurable throughput trade-offs across prefill and decode stages.

The Threat Model: Securing AI Compute in Untrusted Clouds

Traditional cloud inference architectures require tenants to trust multiple layers of the cloud provider's infrastructure stack: the physical machine hardware, firmware, baseboard management controller (BMC), hypervisor, and host kernel. A compromise at any of these layers grants an attacker direct access to DMA engines, memory bus taps, or hypervisor memory scanning tools capable of scraping model weights and prompt context.

+-----------------------------------------------------------------------+
| UNTRUSTED CLOUD HOST (Host OS, Hypervisor, BMC, Cloud Operators)      |
|                                                                       |
|  +---------------------------+       +-----------------------------+  |
|  | CPU Confidential VM (CVM) |       | GPU Hardware Enclave (TEE)  |  |
|  | (AMD SEV-SNP / Intel TDX) |       | (NVIDIA Hopper H100 / H200) |  |
|  |                           |       |                             |  |
|  | +-----------------------+ |       | +-------------------------+ |  |
|  | | Serving Engine (vLLM) | |       | | Tensor Cores & SRAM     | |  |
|  | | Plaintext Context     | |       | | Hardware Firewalls      | |  |
|  | +-----------+-----------+ |       | +------------+------------+ |  |
|  |             |             |       |              |              |  |
|  | +-----------v-----------+ |       | +------------v------------+ |  |
|  | | NVIDIA CVM Driver     | |       | | Encrypted HBM3 VRAM     | |  |
|  | | SPDM / Attestation    | |       | | (AES-256 Inline Engine) | |  |
|  | +-----------+-----------+ |       | +------------+------------+ |  |
|  +-------------|-------------+       +--------------|--------------+  |
|                |                                    |                 |
|                +====== Encrypted PCIe Transport ====+=                |
|                        (AES-256-GCM / IDE Link)                       |
+-----------------------------------------------------------------------+

Confidential computing enforces a zero-trust model with respect to the underlying host. The trusted computing base (TCB) is strictly restricted to:

1. The physical silicon (CPU and GPU hardware).
2. Hardware-enforced root-of-trust microcode.
3. The guest operating system and inference serving runtime inside the enclave.

All external components, including the host hypervisor, peripheral devices, and cloud platform operators, are treated as untrusted adversaries. To maintain privacy, the system must guarantee three cryptographic properties:

• Confidentiality of memory and compute: RAM and VRAM contents cannot be read in plaintext from outside the enclave.
• Integrity of execution: Unauthorized modification of instructions, page tables, or register state triggers immediate execution termination.
• Verifiable identity (Remote Attestation): The hardware provides a cryptographically signed measurement proving that the exact intended software stack is executing on genuine, unmodified silicon before any sensitive data or decryption keys are provisioned.

Hardware Architecture: The Composite CPU-GPU Enclave

Securing large language model inference requires a composite TEE architecture uniting a CPU confidential virtual machine (CVM) with one or more hardware-isolated GPUs.

1. CPU TEE Foundation: AMD SEV-SNP and Intel TDX

The CPU CVM provides the secure control plane for the inference engine. Technologies such as AMD SEV-SNP (Secure Encrypted Virtualization-Secure Nested Paging) and Intel TDX (Trust Domain Extensions) enforce memory encryption using dedicated on-die hardware security processors:

• AES-128/256 Memory Encryption: Physical RAM lines are encrypted at the memory controller using transient keys generated at VM boot, inaccessible to the hypervisor.
• Page Table Integrity (Reverse Map Tables): SEV-SNP and TDX prevent hypervisor-level memory remapping, replay attacks, and memory fault injection by validating guest page table mappings against hardware-managed metadata tables.
• Register State Encryption: CPU register contents are scrubbed and encrypted on every VM exit to prevent hypervisor inspection of intermediate scalar activations.

2. GPU Hardware Enclaves: NVIDIA Hopper Architecture

CPUs alone cannot deliver the parallel throughput required for LLM inference. The NVIDIA Hopper (H100/H200) architecture introduced hardware-native confidential computing capabilities to GPU accelerators.

When Confidential Computing (CC) mode is enabled on an H100:

• Hardware Memory Encryption Engine: High-Bandwidth Memory (HBM3) is transparently encrypted using AES-256-XTS/GCM engines situated directly on the GPU memory controllers.
• Hardware Firewalls: Dedicated silicon firewalls block unauthenticated direct memory access (DMA), peer-to-peer memory reads, and external JTAG debugging interfaces.
• State Scrubbing: Prior to tenant initialization, on-chip firmware performs a deterministic purge of all internal register files, L1/L2 caches, and VRAM blocks.

3. Interconnect Security: The PCIe Transfer Bridge

Data moving between the CPU CVM and the GPU crosses the physical PCIe bus. Because the host hypervisor manages PCIe root complexes and DMA mapping, plaintext transit across the bus is vulnerable to physical interposers and hypervisor snooping.

To secure this link, the system utilizes the PCIe Integrity and Data Encryption (IDE) standard and the Security Protocol and Data Model (SPDM). The NVIDIA CVM driver negotiates shared session keys with the GPU hardware root of trust. Data transferred between host memory and GPU memory passes through unencrypted shared bounce buffers outside the CVM, encrypted using AES-256-GCM before transmission and decrypted upon ingestion by the respective enclave.

4. Multi-GPU Constraints: The NVLink Fabric Gap

A critical operational constraint in modern confidential AI is multi-GPU interconnect handling. On standard NVIDIA HGX H100 systems, inter-GPU NVLink switches lack native line-rate hardware encryption in initial silicon revisions. As detailed in the EnclaveX confidential AI architecture study, running distributed multi-GPU tensor parallelism requires one of two configurations:

• Protected PCIe Mode: Inter-GPU communication is routed over encrypted PCIe links rather than raw NVLink, constraining inter-GPU bandwidth to ~128 GB/s bidirectional per card compared to 900 GB/s on native NVLink.
• Single-Node Trust Domain with Plaintext NVLink: The node attests that NVLink is restricted to a dedicated physical chassis, but cross-GPU links lack bus-level cryptographic isolation, shifting trust to physical data center boundaries.

Native line-rate NVLink encryption is integrated into next-generation Blackwell B200 architectures, resolving this multi-GPU communication bottleneck.

Cryptographic Remote Attestation and Key Release

Confidential inference environments operate under a zero-knowledge trust bootstrap. The inference serving container cannot store proprietary model weights or TLS private keys in its base container image; instead, it must acquire them dynamically via remote attestation.

+-----------------------------------------------------------------------------------------+
|                              REMOTE ATTESTATION WORKFLOW                                |
|                                                                                         |
|  [Guest Enclave (CVM + H100)]                  [Key Broker Service (Trustee KBS)]       |
|              |                                                  |                       |
|  1. Boot TEE & generate nonces                                  |                       |
|  2. Collect CPU Quote (SEV-SNP/TDX)                             |                       |
|  3. Collect GPU RIM Evidence (SPDM)                             |                       |
|              |                                                  |                       |
|              |---- 4. Composite Attestation Request ----------->|                       |
|              |     (CPU Quote + GPU Evidence + Workload Hash)   |                       |
|              |                                                  |                       |
|              |                                        5. Validate Certificate Chain     |
|              |                                           against AMD/Intel & NVIDIA CAs |
|              |                                        6. Match RIM measurements         |
|              |                                           against golden firmware hash   |
|              |                                        7. Evaluate release policy        |
|              |                                                  |                       |
|              |<--- 8. Provision Decryption Keys (KMS/Wrapped) --|                       |
|              |                                                  |                       |
|  9. Decrypt weights in VRAM                                     |                       |
|  10. Start inference serving endpoint                           |                       |
+-----------------------------------------------------------------------------------------+

The attestation lifecycle follows a four-stage process managed by frameworks such as Confidential Containers (CoCo) Trustee:

Step 1: Evidence Collection

Upon initialization, the guest kernel queries the CPU security module to produce an attestation quote containing the cryptographic digest of the initial guest memory state, kernel parameters, and virtual firmware. Concurrently, the NVIDIA driver initiates an SPDM session with the GPU, generating an attestation report that hashes the VBIOS, microcode, driver interface, and CC operational flags.

Step 2: Verification Against Reference Integrity Manifests

The composite evidence is transmitted to an external Attestation Service (AS). The AS verifies the cryptographic signatures against AMD/Intel and NVIDIA hardware root certificate authorities. It compares the measured GPU values against Reference Integrity Manifests (RIM), which are digitally signed vendor manifests representing known-good, uncompromised driver and firmware versions.

Step 3: Key Brokering

If the composite quote matches the tenant's security policy, the Key Broker Service (KBS) establishes an attested TLS session with the enclave. The KBS releases the wrapped decryption keys required to unlock the encrypted model weights and internal certificate authorities.

Step 4: In-Memory Decryption

The serving engine (such as vLLM or Triton) reads the encrypted model weights from remote storage, streams the ciphertext into the CVM, decrypts the layers in enclave memory using the provisioned key, and transfers them across the encrypted PCIe link directly into the GPU's encrypted HBM3 memory.

Serving Performance: Quantifying the TEE Tax

A common misconception is that confidential computing either imposes negligible (<1%) overhead or cripples serving throughput (>50%). Real-world measurements reveal that performance overhead depends entirely on whether a workload is compute-bound or memory-transfer-bound.

+-------------------------------------------------------------------------+
|                  CONFIDENTIAL COMPUTING OVERHEAD PROFILE                |
|                                                                         |
|  Workload Phase      Primary Bottleneck       TEE Performance Penalty   |
|  ---------------------------------------------------------------------  |
|  Model Weight Load   PCIe DMA / Disk Decrypt  +15% to +35% (One-time)   |
|  Attestation Boot    Network / Crypto Verif   1.5s - 3.2s (One-time)    |
|  Prefill (TTFT)      Compute / PCIe Bounce    +4% to +12% (Per request) |
|  Decode (TPOT)       HBM3 Memory Bandwidth    +1.5% to +4% (Per token)  |
|  End-to-End Batch    Mixed Compute / Transfer +3% to +8% (Overall)      |
+-------------------------------------------------------------------------+

As analyzed in recent empirical evaluations by Particula Tech and Mithril Security, confidential serving overhead is partitioned across distinct operational phases:

1. Prefill Phase (Time to First Token)

During prefill, the engine processes large input prompt context in parallel. Because prompts must be transferred from the CPU CVM to the GPU across the PCIe bus via encrypted bounce buffers, memory copying and AES-256-GCM encapsulation introduce latency.

• For small prompt lengths (<512 tokens), PCIe link overhead is prominent, resulting in an 8% to 12% increase in Time to First Token (TTFT).
• For large prompt lengths (>4096 tokens), compute time on the Tensor Cores dominates transfer overhead, shrinking the prefill penalty to 3% to 5%.

2. Decode Phase (Time Per Output Token)

Autoregressive token generation generates one token per step and is bound by GPU memory bandwidth rather than PCIe transfer speed. Once the KV cache is allocated in HBM3, token generation executes entirely within the GPU's internal memory bus.

• The AES-256 inline memory encryption engine on H100 hardware introduces less than 2% latency degradation on standard GEMM and attention kernels.
• Benchmarks for Llama-3 8B and 70B show Time Per Output Token (TPOT) degradation remaining consistently between 1.5% and 4.0% across batch sizes ranging from 1 to 64.

3. End-to-End Serving Throughput

On balanced production workloads (1024 input tokens, 256 output tokens, continuous batching), total throughput loss typically stabilizes between 3% and 7%. The performance penalty rises to 15-20% only when workloads involve frequent CPU-GPU tensor swapping, unoptimized dynamic memory allocation, or cross-node pipeline parallelism lacking line-rate hardware network encryption.

Operational Considerations and Failure Modes

Engineering production-grade confidential inference requires addressing specific infrastructure trade-offs:

• Instance Cold Starts (10s-30s baseline vs 25s-60s in TEE): Attestation handshakes, RIM validation, and wrapped key release add 1.5s to 3.5s of cryptographic overhead before model weight streaming begins. Mitigation: maintain pre-attested warm worker pools in Kubernetes and decouple node attestation from dynamic request routing.
• GPU Telemetry and Profiling Constraints: Standard host-level monitoring tools (such as Nsight Systems, unauthenticated DCGM exporters, and nvtop) cannot access internal GPU counters because the hardware firewall blocks diagnostic DMA. Mitigation: route telemetry exclusively via authenticated in-enclave OpenTelemetry collectors over mutual TLS.
• Memory Allocation and PagedAttention Overhead: PagedAttention dynamic allocations that cross the CPU-GPU boundary require extra bounce-buffer copying and AES encapsulation. Mitigation: pre-allocate contiguous KV cache pools at startup and size CVM host RAM to at least double the target model's parameter footprint.
• Side-Channel and Traffic Analysis Risks: An untrusted cloud hypervisor cannot read memory contents but can monitor packet timing, stream token emission intervals, and memory access patterns. Mitigation: implement fixed-size token streaming padding, constant-time batch execution loops, and oblivious routing proxies.
• Firmware Updates and Policy Drift: Rolling kernel or VBIOS upgrades on host nodes alter cryptographic PCR measurements, causing attestation failures against existing RIM policies. Mitigation: synchronize CI/CD policy deployment with signed vendor RIM bundles and maintain version-pinned enclave deployment pipelines.

Strategic Summary

Confidential LLM inference has shifted from an experimental research domain into a viable enterprise deployment model. With hardware-assisted GPU encryption on NVIDIA Hopper architectures and standardized attestation frameworks like Trustee, organizations can deploy proprietary models and process sensitive customer context on public cloud infrastructure without exposing plaintext data to cloud providers or compromised hypervisors.

The operational overhead is characterized by a one-time cold-start attestation latency (1.5 to 3.5 seconds) and an end-to-end throughput tax of 3% to 7% for standard serving workloads. For mission-critical deployments handling regulated data, this marginal performance cost provides mathematically verifiable isolation and rigorous defense-in-depth across the AI infrastructure lifecycle.

Sources

• EnclaveX: End-to-End Confidential AI with CPU/GPU TEEs (arXiv:2606.31408)
• NVIDIA Hopper Architecture: Confidential Computing on H100 GPUs
• NVIDIA Confidential Containers Architecture & Remote Attestation
• Confidential Containers (CoCo) Trustee Architecture
• Particula Tech: Confidential GPU Inference and H100 Performance Tax
• Mithril Security: Choosing the Best GPU Stack for Confidential LLM Deployment
• AMD SEV-SNP Architecture & Technical Overview
• Intel Trust Domain Extensions (Intel TDX) Specification
• DMTF Security Protocol and Data Model (SPDM) Standard