Document Parsing and Visual Retrieval for Production RAG: Architecture, Benchmarks, and Serving Trade-Offs for Docling, Marker, MinerU, and ColPali

The retrieval quality of a Retrieval-Augmented Generation (RAG) system is strictly bounded by the fidelity of its document ingestion pipeline. In enterprise environments, the vast majority of domain knowledge remains locked in unstructured Portable Document Format (PDF) files, scanned reports, technical manuals, and multi-column research papers.

Naive text extraction libraries (such as PyPDF or pdfminer) read PDF content streams in raw serialization order. This process routinely interleaves parallel text columns, erases table grid coordinates, strips mathematical equations, and mixes running headers with body copy. When corrupted text is fed downstream into chunking algorithms and embedding models, dense vector retrieval fails and generation models hallucinate.

Modern document processing architectures have bifurcated into two distinct operational paradigms:

1. Pipeline Layout Parsing: Specialized vision and layout models parse page topology, extract bounding boxes, isolate tables and formulas, and reconstruct a semantic markdown or structured Abstract Syntax Tree (AST).
2. End-to-End Multimodal Visual Retrieval: Vision-Language Models (VLMs) embed rasterized page images directly into multi-vector representations, eliminating OCR and layout reconstruction from the retrieval path.

Here is a technical analysis of the architectural trade-offs, benchmark performance, and deployment patterns across the leading open-source frameworks: IBM Docling, Marker, OpenDataLab MinerU, and ColPali.

1. Pipeline Layout Parsing: Architectural Breakdown

Layout-aware parsers process documents through multi-stage neural pipelines. Rather than treating a PDF as an unstructured character stream, they treat each page as a 2D visual canvas, applying computer vision models to identify semantic blocks before extracting and ordering text.

IBM Docling: The Structured Data Model Approach

Developed by IBM Research and licensed permissively under MIT, Docling is engineered as an enterprise-grade document conversion library. Its architecture is built around a centralized intermediate representation called DoclingDocument, which preserves reading order, hierarchy, bounding coordinates, and metadata.

Key architectural components include:

• Layout Analysis: Docling uses an RT-DETR (Real-Time DEtection TRansformer) model trained on the DocLayNet benchmark dataset. It classifies bounding boxes into structural categories: text paragraphs, title headers, tables, figures, code blocks, lists, and formulas.
• Table Extraction via TableFormer: Rather than relying on heuristic line detection, Docling integrates TableFormer, a vision transformer architecture designed for table structure recognition. TableFormer achieves over 91% Tree Edit Distance Similarity (TEDS) on complex benchmarks like FinTabNet, reconstructing row and column spans, headers, and cell relationships even in borderless financial tables.
• Hybrid Text Resolution: Docling extracts programmatic text layers directly from digital PDFs to avoid OCR distortion. For scanned pages or image regions, it falls back to OCR backends (EasyOCR, Tesseract, or RapidOCR).
• Execution Footprint: Docling is optimized for commodity hardware. On x86 CPUs without GPU acceleration, Docling averages 3.1 seconds per page, and 1.27 seconds per page on Apple Silicon (M3 Max), making it viable for environments without dedicated GPU clusters.

Marker: Throughput Optimization and Selective OCR

Created by Vik Paruchuri and maintained by Datalab, Marker is designed for high-throughput batch conversion of books, papers, and complex documents into clean Markdown and JSON.

In its v2.0 architecture, Marker optimizes inference speed through selective execution:

• Selective OCR via pdftext: Marker uses a fast C-backed text extraction engine (pdftext). It inspects character encoding confidence and only routes corrupted, rotated, or scanned regions to visual neural models.
• Surya Model Suite: When visual parsing is required, Marker deploys the Surya model family for layout analysis, reading order sorting, and OCR in 90+ languages.
• Multi-Mode Execution: Marker supports configurable operational modes:
• balanced: Uses Surya VLM for layout detection and targeted OCR, reaching 76.0% on the olmOCR benchmark.
• fast: Employs a lightweight 20M-parameter layout model with minimal VLM intervention.
• --disable_ocr: Bypasses visual neural networks entirely, running pure text-layer extraction on CPU at up to 23.7 pages per second.
• Inference Server Architecture: Multiple lightweight CPU workers share a single centralized VLM inference backend (hosted on vLLM for NVIDIA GPUs or llama.cpp for CPU environments), decoupling worker scaling from GPU VRAM allocation.

OpenDataLab MinerU: Scientific Literature and Formula Fidelity

Developed by OpenDataLab (the team behind InternLM), MinerU focuses on high-precision parsing of academic literature, technical whitepapers, and scientific documentation containing complex formulas and East Asian languages.

MinerU's pipeline leverages the PDF-Extract-Kit model suite:

• Layout Detection: Integrates DocLayout-YOLO for region bounding box segmentation.
• Mathematical Formula Recognition via UniMERNet: Mathematical formulas (both inline and display blocks) are detected and routed to UniMERNet, a specialized neural network that transcribes mathematical notation directly into standardized LaTeX syntax.
• High-Throughput CUDA Acceleration: On dedicated GPU hardware (such as an NVIDIA L4), MinerU achieves a processing speed of 0.21 seconds per page, outperforming traditional pipelines in batch throughput when VRAM is plentiful.
• Trade-Offs: MinerU requires a heavy dependency footprint and complex environment configuration. CPU performance is slower (~3.3 seconds per page), and macOS execution remains constrained compared to Linux CUDA environments.

2. Multimodal Visual Retrieval: The ColPali Paradigm

While pipeline parsers convert visual documents into textual Markdown for standard dense embeddings, ColPali (Contextualized Late Interaction Over PaliGemma) introduces an alternative paradigm: visual document retrieval without parsing.

+-------------------------------------------------------------------+
|                  PARADIGM 1: PIPELINE PARSING                     |
| PDF -> Layout Detector -> OCR/TableFormer -> Markdown -> Embeddings|
+-------------------------------------------------------------------+

+-------------------------------------------------------------------+
|                  PARADIGM 2: VISUAL RETRIEVAL                     |
| PDF -> Render Page Image -> PaliGemma VLM -> Multi-Vector Index   |
+-------------------------------------------------------------------+

Architectural Mechanics of ColPali

Instead of extracting text, ColPali renders PDF pages as images (typically at 448x448 resolution) and feeds them into a Vision-Language Model backbone:

1. Vision Transformer (SigLIP-So400m): Splits the page image into spatial patches (e.g., 32x32 grid) and generates visual patch embeddings.
2. Language Model Projection (Gemma 2B): Linear projection layers map the visual patch tokens into the language model's embedding space. The transformer layers contextualize visual elements (diagrams, flowcharts, table lines) alongside textual typography.
3. Low-Dimensional Multi-Vector Output: Each page is represented as a collection of patch vectors (typically 1,024 tokens) projected down to dimension $D = 128$.
4. Late Interaction Scoring (MaxSim): When a user submits a textual query, the query tokens are embedded into the same 128-dimensional space. Retrieval scoring uses the ColBERT late-interaction formula:

$$\text{Score}(Q, D) = \sum_{q_i \in Q} \max_{d_j \in D} (q_i \cdot d_j^T)$$

For each query token $q_i$, the system computes the maximum cosine similarity across all document patch vectors $d_j$, summing the maximums to produce the final relevance score.

Advantages and System Constraints of Visual Retrieval

• Bypassing OCR Bottlenecks: ColPali completely avoids OCR errors, font decoding failures, and multi-column ordering bugs. Figures, bar charts, organizational charts, and infographic layouts are indexed directly from visual geometry.
• Benchmark Performance: On the ViDoRe (Visual Document Retrieval) benchmark, ColPali outperforms standard dense text retrievers (such as BM25 + BGE-M3) by over 15 points in nDCG@10 on visually rich document collections.
• Storage and Memory Overhead: While a standard dense vector requires 1,536 float32 dimensions (~6 KB per chunk), ColPali requires 1,024 vectors of 128 float32 dimensions (~512 KB per page). Scaling this to millions of pages requires specialized vector engines (such as Qdrant or Vespa) with scalar quantization and binary compression.
• Context Generation Bottleneck: ColPali returns page images, not serialized text strings. To provide context to a downstream text-only LLM, the system must either run a second-stage visual LLM (such as GPT-4o or Gemini Flash) on the retrieved page image or maintain a parallel text index.

3. Comparative Benchmarks and Hardware Profiles

Selecting an ingestion engine requires balancing processing speed, memory consumption, table accuracy, and license compatibility.

Structural Performance Comparison

• IBM Docling:
• License: MIT
• Primary Strength: Enterprise table extraction (TableFormer >91% TEDS), structured DoclingDocument data model, multi-format input (PDF, DOCX, PPTX, HTML).
• Inference Speed: ~3.1 s/page (CPU), ~0.49 s/page (NVIDIA L4 GPU).
• Best For: Regulated enterprise RAG, financial reports (10-K filings), complex tables, and LlamaIndex/LangChain pipelines.
• Marker (v2.0):
• License: GPL-3.0 / Commercial tier (Datalab)
• Primary Strength: High-throughput Markdown generation, selective OCR caching, decoupled client-server architecture.
• Inference Speed: ~16.0 s/page (CPU with OCR), ~0.86 s/page (NVIDIA L4 GPU), up to 23.7 pages/s on CPU (no-OCR mode).
• Best For: High-volume document archives, bulk book conversions, and clean Markdown chunking pipelines.
• OpenDataLab MinerU:
• License: Apache-2.0
• Primary Strength: Mathematical formula transcription (UniMERNet LaTeX), CJK and Latin multilingual OCR, DocLayout-YOLO segmentation.
• Inference Speed: ~3.3 s/page (CPU), ~0.21 s/page (NVIDIA L4 GPU).
• Best For: Scientific literature (arXiv corpora), technical whitepapers, and math-dense academic RAG.
• ColPali:
• License: Apache-2.0 (weights subject to PaliGemma base license)
• Primary Strength: Native visual retrieval over charts, infographics, tables, and slide decks without layout parsing.
• Inference Speed: ~0.15 s/page (indexing embedding on A100/H100 GPU); sub-50ms query late-interaction scoring.
• Best For: Slide decks, scanned historical documents, graphic-heavy manuals, and visual search applications.

4. Production Architectural Patterns

To balance accuracy, compute costs, and latency, production enterprise RAG systems increasingly deploy hybrid ingestion patterns rather than relying on a single tool.

                           [ Ingestion Stage ]
                                    |
          +-------------------------+-------------------------+
          |                                                   |
  [ Text-Dense Documents ]                            [ Visual-Dense Slides & Charts ]
  (10-K, Contracts, Policies)                         (Infographics, Manuals, Decks)
          |                                                   |
   [ IBM Docling ]                                      [ ColPali Indexing ]
  (TableFormer + RT-DETR)                             (Multi-Vector Patch Index)
          |                                                   |
   [ Markdown / JSON ]                                        |
          |                                                   |
   [ Hybrid Search Index ]                                    |
   (pgvector / BM25 / SPLADE)                                 |
          |                                                   |
          +-------------------------+-------------------------+
                                    |
                           [ Query Routing ]
                                    |
                       [ Retrieval & Generation ]

Pattern A: Single-Stage Layout Extraction with Docling

For text-centric enterprise repositories (legal contracts, corporate policies, regulatory disclosures), a single-stage parsing pipeline using Docling provides the cleanest integration:

1. Ingestion: Docling parses the incoming PDF, executes TableFormer on detected table regions, and outputs a semantic DoclingDocument.
2. Hierarchical Chunking: Chunks are split along section boundaries, headers, and discrete table structures, preserving structural context in chunk metadata.
3. Indexing: Chunks are embedded with standard dense models (such as BGE-large or text-embedding-3-large) and indexed into a vector database alongside BM25 sparse tokens.
4. Generation: The retrieved Markdown tables and text chunks are injected directly into the LLM system prompt.

Pattern B: Two-Stage Visual-Text Hybrid Routing

For corpora with heavy mixtures of graphical diagrams, presentation slides, and dense text, a two-stage hybrid architecture achieves the highest end-to-end recall:

1. First-Stage Visual Discovery (ColPali): ColPali indexes full-page images in a multi-vector store (such as Qdrant). At query time, MaxSim late interaction retrieves the top 5 to 10 most relevant page images, accurately surfacing visual charts and tables that standard text retrievers miss.
2. Second-Stage Targeted Extraction or VLM Synthesis:

• If the generation model is a native multimodal LLM (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro), the top retrieved page images are passed directly to the vision input of the model.
• If the generation model is text-only or cost-constrained, targeted Docling/Marker parsers process only the retrieved candidate pages on demand, converting them into structured Markdown for LLM prompt context.

This hybrid approach prevents the prohibitive compute cost of running heavy OCR and layout pipelines across millions of irrelevant archive pages while preserving visual comprehension during search.

Sources

• Faysse, M., Sibille, H., Wu, T., Omrani, B., Viaud, G., Hudelot, C., & Colombo, P. (2024). ColPali: Efficient Document Retrieval with Vision Language Models. arXiv:2407.01449. https://arxiv.org/abs/2407.01449
• IBM Research. (2025). Docling: An Efficient Open-Source Toolkit for AI-driven Document Conversion. arXiv:2501.17887. https://arxiv.org/abs/2501.17887
• Wang, B., Xu, C., Zhao, X., et al. (2024). MinerU: An Open-Source Solution for Precise Document Content Extraction. arXiv:2409.18839. https://arxiv.org/abs/2409.18839
• Wang, B., Gu, Z., Xu, C., et al. (2024). UniMERNet: A Universal Network for Real-World Mathematical Expression Recognition. arXiv:2404.15254. https://arxiv.org/abs/2404.15254
• Paruchuri, V. (2024). Marker: High-Accuracy Document to Markdown Conversion. GitHub Repository. https://github.com/datalab-to/marker
• Hugging Face. (2024). ColPali: Efficient Document Retrieval with Vision Language Models. Hugging Face Blog. https://huggingface.co/blog/manu/colpali