Asynchronous Batch Inference in Production: Architecture, Queue Scheduling, and Cost Arbitrage
Interactive AI applications require low Time-to-First-Token (TTFT) and high inter-token generation speed to maintain responsive user experiences. Achieving sub-second latency targets forces infrastructure teams to overprovision GPU capacity to absorb peak demand spikes. However, non-interactive production workloads (such as historical document processing, embedding generation, nightly model evaluation suites, synthetic data generation, and offline classification) do not require real-time execution.
Treating background workloads as synchronous HTTP requests creates two operational bottlenecks: severe rate-limiting contention against user-facing traffic and inflated inference spend. Asynchronous batch inference architectures resolve these challenges by exploiting provider cost arbitrage (50% discounts) and hardware-level compute saturation.
1. The Economics of Provider Batch APIs
Major foundation model providers, including OpenAI, Anthropic, and Google Cloud, offer dedicated asynchronous batch endpoints that provide a flat 50% discount on both input and output token pricing in exchange for a 24-hour completion window.
Provider Mechanics and Constraints
- OpenAI Batch API (
/v1/batches): - Discount: 50% discount on standard prompt and completion token rates.
- Turnaround Window: 24-hour SLA window (median execution time under 1 hour).
- Payload Limits: Up to 50,000 requests or 200 MB per JSONL file via the Files API.
- Rate Quota: Operates on a dedicated batch pool with significantly higher TPM/RPM limits.
- Prefix Optimization: Automatic prefix matching against prompt cache entries.
- Anthropic Message Batches (
/v1/messages/batches): - Discount: 50% discount across all Claude models (input, output, and cache operations).
- Turnaround Window: 24-hour SLA window (median execution time under 1 hour).
- Payload Limits: Up to 100,000 requests or 256 MB per batch payload.
- Rate Quota: Separate high-throughput concurrency queue.
- Prefix Optimization: Supports explicit
cache_controlblocks with compounding discounts. - Google Gemini Batch Prediction (Vertex AI):
- Discount: 50% discount on standard Gemini model inference.
- Turnaround Window: 24-hour SLA window (median execution time under 2 hours).
- Payload Limits: Up to 2 GB per batch file stored in Google Cloud Storage.
- Rate Quota: Managed via dedicated project quota pools.
- Prefix Optimization: Integrates with Vertex AI Context Caching.
Providers offer this pricing discount because asynchronous batches act as valley-filling compute. Datacenters maintain fixed GPU allocations to handle peak daytime traffic. When diurnal demand falls, idle Tensor Cores run non-real-time batch queues without increasing baseline capital expenditure.
Independent Rate Limits
Synchronous endpoints enforce strict Requests Per Minute (RPM) and Tokens Per Minute (TPM) limits to prevent server starvation. A single bulk backfill can exhaust an organization's TPM quota, triggering HTTP 429 errors for real-time customer sessions. Batch APIs route requests to a distinct scheduler with independent, orders-of-magnitude larger volume pools (often tens of millions of tokens per batch job), isolating batch execution from production traffic.
2. Hardware Economics: Memory-Bound Decode vs. Compute-Bound Batching
The economic viability of batch inference extends directly to self-hosted LLM infrastructure. The fundamental constraint of autoregressive transformer inference is the arithmetic intensity of the generation loop.
+-------------------------------------------------------------------------+
| Online Serving (Batch Size 1-8): Memory-Bandwidth Bound |
| - Weights loaded from HBM to SRAM for every single token. |
| - Low Arithmetic Intensity (< 1 FLOP/byte). |
| - Tensor Core Model FLOPs Utilization (MFU): 15% - 25%. |
+-------------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------------+
| Offline Batch Inference (Batch Size 256-1024): Compute Bound |
| - Weights loaded once; reused across hundreds of sequence vectors. |
| - High Arithmetic Intensity (Approaching Hardware Roofline). |
| - Tensor Core Model FLOPs Utilization (MFU): 70% - 90%. |
+-------------------------------------------------------------------------+The Roofline Bottleneck in Autoregressive Serving
During low-batch online serving (batch size 1 to 8), token generation performs General Matrix-Vector (GEMV) multiplications. For every token emitted by a 70B parameter FP16 model, the GPU must transfer 140 GB of model weights from High Bandwidth Memory (HBM) into on-chip SRAM:
On an NVIDIA H100 SXM (3.35 TB/s memory bandwidth), reading 140 GB of weights takes ~41.8 milliseconds per step, generating ~24 tokens per second for a single stream. The GPU's 989 TFLOPS of FP16 compute capacity remains largely idle, resulting in Model FLOPs Utilization (MFU) below 20%.
Saturated Batch GEMM Scaling
In offline batch inference runtimes such as the vLLM Offline LLM Engine or Ray Data LLM, requests are batched into dense matrices (batch size 256 to 1024). Matrix operations transition to compute-bound General Matrix-Matrix (GEMM) multiplications:
- Weight Reuse: Model weights are loaded into SRAM once and multiplied against hundreds of prompt vectors simultaneously.
- Compute Saturation: Arithmetic intensity rises by orders of magnitude, pushing GPU execution into the compute-bound regime of the roofline model.
- Throughput Density: An H100 processing an 8B parameter model in offline batch mode can exceed 30,000 input tokens/second during prefill and 2,500+ generation tokens/second during decode, delivering up to 5x higher token throughput per dollar compared to dynamic online serving.
3. Stacking Reductions: Prompt Caching Inside Batches
The cost advantages of batch APIs compound when paired with prompt caching. Many high-volume batch workloads evaluate multiple records against identical instructions, such as applying a 4,000-token system prompt and few-shot rubric across 50,000 distinct customer tickets.
Batch Request Line 1: [Shared System Prompt (4,000 tokens)] + [Item 1 (200 tokens)]
Batch Request Line 2: [Shared System Prompt (4,000 tokens)] + [Item 2 (180 tokens)]
...
Batch Request Line N: [Shared System Prompt (4,000 tokens)] + [Item N (220 tokens)]When structuring batch files for providers supporting prompt caching (such as Anthropic Prompt Caching), the pricing benefits stack:
- Standard Prompt Token Cost: Base rate.
- Batch Processing: 50% discount on base rate.
- Cache Read: 90% discount on prompt token cost.
- Stacked Batch + Cache Read: 50% applied to the cached rate, yielding a net 95% reduction on recurring input tokens.
For a dataset of 50,000 items sharing a 4,000-token context, input token costs drop from $600 to $30, while eliminating redundant attention computations during provider execution.
4. Production Batch Pipeline Architecture
Building a reliable batch processing pipeline requires robust state management, request chunking, error isolation, and idempotent data reconciliation.
+------------------+ +-------------------+ +-------------------+
| Data Lake / DB | ---> | JSONL Chunker & | ---> | Provider Files / |
| (Source Table) | | Custom ID Inject | | S3 Staging Bucket |
+------------------+ +-------------------+ +-------------------+
|
v
+------------------+ +-------------------+ +-------------------+
| Relational DB / | <--- | Error Parser & | <--- | Batch Execution |
| Vector Store | | Retry Classifier | | State Orchestrator|
+------------------+ +-------------------+ +-------------------+Architectural Pillars
1. Deterministic Custom ID Correlation
Batch API responses are returned out of order relative to the input file. To guarantee deterministic state synchronization, every line in the input JSONL must contain an immutable custom_id encoding job context:
{"custom_id": "job_982f:row_10482:chunk_0", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "Analyze sentiment:..."}]}}The consumer parses custom_id on ingestion to update the corresponding primary key in relational storage without holding in-memory tracking arrays.
2. Bounded File Chunking
Providers enforce strict file size (200 MB to 256 MB) and record count (50,000 to 100,000 requests) limits per batch. Ingestion workers must chunk continuous data streams into bounded slices, generating checksums (SHA-256) for each chunk prior to upload.
3. State Machine Orchestration
Using durable workflow engines (e.g., Temporal, Celery, or AWS Step Functions), batch jobs transition through explicit states:
[QUEUED] -> [FILE_UPLOADED] -> [BATCH_SUBMITTED] -> [POLLING_STATUS] -> [DOWNLOADING_OUTPUT] -> [RECONCILED]
|
+--> [FAILED / EXPIRED] -> [DLQ_TRIGGER]4. Error Isolation and Partial Retries
Batch jobs can partially succeed. If 49,950 requests succeed and 50 fail due to transient server issues (HTTP 500) or token context overflows, providers generate an error_file_id. The orchestrator must parse both the output file and error file, recording successful completions while routing failed rows to a Dead Letter Queue (DLQ) for automated re-batching or fallback to synchronous execution.
5. Production Implementation Blueprint
The following Python implementation demonstrates a production-grade batch orchestrator using the OpenAI SDK, incorporating file preparation, batch creation, exponential backoff polling, and error separation:
import json
import time
from pathlib import Path
from openai import OpenAI
client = OpenAI()
def build_batch_file(records: list[dict], output_path: Path) -> Path:
"""Formats raw records into an OpenAI-compatible JSONL batch file."""
with open(output_path, "w", encoding="utf-8") as f:
for record in records:
payload = {
"custom_id": f"rec_{record['id']}",
"method": "POST",
"url": "/v1/chat/completions",
"body": {
"model": "gpt-4o-mini",
"temperature": 0.2,
"max_tokens": 500,
"messages": [
{"role": "system", "content": "Extract structured entities in JSON format."},
{"role": "user", "content": record["text"]}
],
"response_format": {"type": "json_object"}
}
}
f.write(json.dumps(payload) + "\n")
return output_path
def run_batch_job(batch_file_path: Path) -> dict:
"""Uploads file, creates batch job, and polls to completion."""
# Step 1: Upload file to Files API
with open(batch_file_path, "rb") as file_stream:
batch_file = client.files.create(file=file_stream, purpose="batch")
# Step 2: Create batch job
batch = client.batches.create(
input_file_id=batch_file.id,
endpoint="/v1/chat/completions",
completion_window="24h",
metadata={"job_type": "entity_extraction", "source_file": str(batch_file_path.name)}
)
# Step 3: Poll status with exponential backoff
delay = 15
terminal_states = {"completed", "failed", "expired", "cancelled"}
while True:
status_obj = client.batches.retrieve(batch.id)
current_status = status_obj.status
if current_status in terminal_states:
return {
"batch_id": batch.id,
"status": current_status,
"output_file_id": status_obj.output_file_id,
"error_file_id": status_obj.error_file_id,
"request_counts": {
"total": status_obj.request_counts.total,
"completed": status_obj.request_counts.completed,
"failed": status_obj.request_counts.failed
}
}
time.sleep(delay)
delay = min(delay * 1.5, 300)
def retrieve_and_parse_results(output_file_id: str) -> list[dict]:
"""Downloads and deserializes batch output records."""
file_content = client.files.content(output_file_id).text
results = []
for line in file_content.strip().split("\n"):
if line:
record = json.loads(line)
custom_id = record["custom_id"]
response_body = record["response"]["body"]
extracted_json = json.loads(response_body["choices"][0]["message"]["content"])
results.append({"custom_id": custom_id, "data": extracted_json})
return results6. Architecture Selection Framework
Choosing between synchronous endpoints, managed cloud batch APIs, and self-hosted offline batch clusters depends on three operational parameters:
- Synchronous API Endpoints:
- Best For: User-facing interactive applications and conversational agents.
- Latency Profile: Sub-second to under 3 seconds per response.
- Cost Baseline: Standard 100% token list pricing.
- Infrastructure Burden: Low (stateless HTTP calls).
- Isolation: Shared multi-tenant rate limits.
- Cloud Managed Batch APIs (OpenAI / Anthropic / Gemini):
- Best For: Asynchronous ingestion, nightly evals, and medium-scale bulk transformations.
- Latency Profile: 15 minutes to 24 hours (queue bounded).
- Cost Baseline: 50% discount on input and output tokens.
- Infrastructure Burden: Low (provider manages execution, storage, and retries).
- Isolation: High token rate limit headroom decoupled from real-time keys.
- Self-Hosted Offline Batch Engines (vLLM / Ray Data):
- Best For: High-volume private enterprise datasets (>1B tokens/month) and strict data residency requirements.
- Latency Profile: Minutes to hours based on local cluster provisioning.
- Cost Baseline: Amortized GPU instance rental costs ($/hour).
- Infrastructure Burden: High (Kubernetes, distributed storage, GPU driver and kernel management).
- Isolation: Full enterprise VPC perimeter isolation.
