Sigmoid Loss for Language Image Pre-Training (SigLIP): Mathematical Foundations, Pairwise Binary Classification, Softmax Normalization Removal, and Batch Size Scaling in Multimodal Vision-Language Models

Executive Summary

Multimodal foundation models rely heavily on contrastive representation learning to map visual features and natural language tokens into a unified embedding space. For years, the dominant paradigm was the InfoNCE objective popularized by CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021), which uses categorical cross-entropy with a softmax normalization over all paired elements in a mini-batch. While effective, the softmax normalization imposes a severe distributed systems bottleneck: computing the normalization denominator requires gathering representations across all accelerator nodes, coupling memory complexity quadratically to batch size ($O(B^2)$) and destabilizing optimization when mini-batches are too small or excessively large.

Sigmoid Loss for Language Image Pre-Training (SigLIP), introduced by Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer at Google DeepMind (ICCV 2023), replaces the global softmax contrastive loss with an independent pairwise binary logistic regression loss. By treating every image-text pair in a batch as an independent binary classification problem (positive if paired, negative if unpaired), SigLIP eliminates the cross-device normalization denominator, decouples communication overhead, and stabilizes optimization across arbitrary batch sizes. Today, SigLIP and its successor SigLIP 2 serve as the standard visual backbone for leading open-weight multimodal models, including Google's PaliGemma and PaliGemma 2, Gemma 2 vision variants, LLaVA-NeXT, and DeepSeek-VL.

The InfoNCE Bottleneck in Standard CLIP

In standard contrastive language-image pre-training, an image encoder $f_\theta$ and a text encoder $g_\phi$ map an image $I_i$ and a text description $T_j$ into normalized $d$-dimensional embedding vectors $\mathbf{u}_i = f_\theta(I_i) / \|f_\theta(I_i)\|_2$ and $\mathbf{v}_j = g_\phi(T_j) / \|g_\phi(T_j)\|_2$.

Given a mini-batch of $B$ paired examples $\{(I_i, T_i)\}_{i=1}^B$, the CLIP objective calculates pairwise cosine similarities $s_{ij} = \mathbf{u}_i^\top \mathbf{v}_j$ scaled by a learnable temperature parameter $\tau = 1/t$. The loss is the symmetric sum of two cross-entropy terms: image-to-text ($\mathcal{L}_{\text{i2t}}$) and text-to-image ($\mathcal{L}_{\text{t2i}}$):

$$\mathcal{L}_{\text{CLIP}} = \frac{1}{2B} \sum_{i=1}^B \left( -\log \frac{\exp(s_{ii} / \tau)}{\sum_{j=1}^B \exp(s_{ij} / \tau)} \right) + \frac{1}{2B} \sum_{j=1}^B \left( -\log \frac{\exp(s_{jj} / \tau)}{\sum_{i=1}^B \exp(s_{ij} / \tau)} \right)$$

This formulation exhibits three major architectural limitations:

1. Global Denominator Coupling: The denominator $\sum_{j=1}^B \exp(s_{ij} / \tau)$ requires a global sum across all $B$ samples. In large-scale distributed training across hundreds of GPUs or TPUs, computing this denominator mandates an expensive all-gather collective communication step across all ranks before computing the softmax partition function.
2. Batch Size Dependency for Negative Quality: Under softmax cross-entropy, the probability assigned to the true positive depends strictly on the competitive sum of all negatives in the batch. If batch size $B$ is small (e.g., $B \le 4096$), the denominator provides insufficient negative contrast, causing poor representation quality. Consequently, CLIP required scaling batch size to 32,768 or 65,536 to achieve strong zero-shot performance.
3. Memory Footprint and Redundant Gradients: The full $B \times B$ logit matrix must be instantiated and retained in memory during the backward pass to compute softmax gradients, consuming significant high-bandwidth memory (HBM) on training accelerators.

Mathematical Formulation of SigLIP

SigLIP reframes multimodal contrastive learning by discarding the multinomial distribution over the batch entirely. Instead, every entry $(i, j)$ in the $B \times B$ pairwise similarity matrix is treated as an independent binary classification event.

Let $z_{ij} \in \{-1, +1\}$ denote the binary ground-truth label for the pair $(I_i, T_j)$:

$$z_{ij} = \begin{cases} +1 & \text{if } i = j \quad (\text{positive paired example}) \\ -1 & \text{if } i \neq j \quad (\text{negative unpaired example}) \end{cases}$$

The affine similarity score $s_{ij}$ is computed using a learnable positive scale parameter $t > 0$ (inverse temperature) and a learnable scalar bias $b$:

$$s_{ij} = t \cdot (\mathbf{u}_i^\top \mathbf{v}_j) + b$$

The pairwise SigLIP loss over the entire mini-batch is defined as the mean negative log-sigmoid:

$$\mathcal{L}_{\text{SigLIP}} = -\frac{1}{B} \sum_{i=1}^B \sum_{j=1}^B \log \sigma\left( z_{ij} \cdot s_{ij} \right) = -\frac{1}{B} \sum_{i=1}^B \sum_{j=1}^B \log \left( \frac{1}{1 + \exp\left( -z_{ij} \cdot (t \cdot \mathbf{u}_i^\top \mathbf{v}_j + b) \right)} \right)$$

Expanding this expression separates the contribution of the $B$ diagonal positive pairs from the $B(B-1)$ off-diagonal negative pairs:

$$\mathcal{L}_{\text{SigLIP}} = \frac{1}{B} \sum_{i=1}^B \log(1 + \exp(-t \mathbf{u}_i^\top \mathbf{v}_i - b)) + \frac{1}{B} \sum_{i=1}^B \sum_{j \neq i}^B \log(1 + \exp(t \mathbf{u}_i^\top \mathbf{v}_j + b))$$

Notice the normalization factor: the total loss is normalized by $1/B$ rather than $1/B^2$. As batch size $B$ scales, the number of negative pairs per sample grows linearly ($B-1$), naturally increasing the repulsive pressure without diluting the per-example gradient magnitude.

Gradient Dynamics and the Role of Learnable Bias

To understand why SigLIP trains stably from scratch, we examine the gradient of the pairwise loss $\ell_{ij} = -\log \sigma(z_{ij} s_{ij})$ with respect to the pre-activation similarity $s_{ij}$:

$$\frac{\partial \ell_{ij}}{\partial s_{ij}} = -z_{ij} \cdot (1 - \sigma(z_{ij} s_{ij})) = \begin{cases} - (1 - \sigma(s_{ij})) = - \sigma(-s_{ij}) & \text{for } z_{ij} = +1 \\ + \sigma(s_{ij}) & \text{for } z_{ij} = -1 \end{cases}$$

For the normalized image embedding $\mathbf{u}_i$, the chain rule gives:

$$\frac{\partial \mathcal{L}_{\text{SigLIP}}}{\partial \mathbf{u}_i} = -\frac{t}{B} \left( \sigma(-s_{ii}) \mathbf{v}_i - \sum_{j \neq i}^B \sigma(s_{ij}) \mathbf{v}_j \right)$$

This reveals two crucial dynamics:

• Attractive Force: The positive target $\mathbf{v}_i$ pulls $\mathbf{u}_i$ with magnitude proportional to $\sigma(-s_{ii}) = 1 - \sigma(s_{ii})$. When alignment is weak ($s_{ii} \ll 0$), the gradient multiplier approaches 1.0; as alignment perfects ($s_{ii} \gg 0$), the attractive force smoothly decays to zero.
• Repulsive Force: Each negative $\mathbf{v}_j$ repels $\mathbf{u}_i$ with magnitude proportional to $\sigma(s_{ij})$. If a negative pair is already orthogonal or dissimilar ($s_{ij} \ll 0$), $\sigma(s_{ij}) \approx 0$, producing virtually zero gradient and preventing gradient noise from uninformative negatives.

The Negative Imbalance Problem and Initialization

In a mini-batch of size $B$, there is 1 positive pair and $B - 1$ negative pairs for each image. If temperature and bias were initialized neutrally ($t=1, b=0$), random embeddings ($\mathbf{u}_i^\top \mathbf{v}_j \approx 0$) would yield $s_{ij} \approx 0$, causing $\sigma(s_{ij}) \approx 0.5$. The net gradient for each sample would be dominated by $(B-1) \times 0.5$ repulsive steps, triggering violent gradient surges that destroy initial weights.

Zhai et al. resolved this imbalance by introducing the learnable bias parameter $b$ with a strongly negative initialization:

• Initial bias: $b = -10$
• Initial temperature: $t = \exp(t')$ with $t' = \log(10) \approx 2.3026$ (yielding $t = 10$)

With $b = -10$ at initialization, the negative pair probability is $\sigma(0 - 10) \approx 4.54 \times 10^{-5}$. Even with a batch size of $B = 32,768$, the cumulative negative gradient per sample is $(32767) \times (4.54 \times 10^{-5}) \approx 1.48$, perfectly balancing the single positive gradient $\sigma(10) \approx 0.9999$. As training progresses, the model automatically tunes $t$ and $b$ to maintain optimal discriminative calibration.

Distributed Systems and Memory Efficiency

Removing the softmax partition function fundamentally alters how contrastive vision-language models scale across distributed hardware clusters.

Elimination of All-Reduce Communication

Under InfoNCE, computing $\sum_{j=1}^B \exp(s_{ij}/t)$ requires gathering all text embeddings across all accelerator devices. In a data-parallel setup with $P$ workers, each worker holds local batch size $b_{\text{local}} = B / P$. To compute the softmax, workers must execute an AllGather operation of size $P \times b_{\text{local}} \times d$, followed by distributed softmax reduction.

Under SigLIP, because each $(i, j)$ loss term is strictly local to its pair, the similarity matrix can be computed in tiled chunks (chunked dot-products) and accumulated asynchronously without requiring global synchronization barriers for the denominator.

Peak Memory Footprint

In standard CLIP implementations, the $B \times B$ logit matrix must be materialized in memory to compute row-wise and column-wise softmax distributions. For a batch size of $B = 65,536$, storing float32 logits requires:

$$\text{Memory} = 65536 \times 65536 \times 4 \text{ bytes} \approx 17.18 \text{ GB}$$

In SigLIP, because binary cross-entropy operates elementwise, the loss and backward pass can be computed on-the-fly inside fused GPU/TPU kernels in micro-blocks of $1024 \times 1024$, reducing activation memory from gigabytes to megabytes.

Empirical Benchmarks and Batch Size Scaling

In their comprehensive ablation across the WebLI (Web-Language-Image) dataset, Zhai et al. demonstrated that SigLIP consistently outperforms standard InfoNCE across nearly all operational regimes:

• Small Batch Performance: At a modest batch size of $B = 4,096$, SigLIP achieved a zero-shot top-1 ImageNet accuracy of 73.1%, whereas standard softmax CLIP achieved only 69.8%. Softmax CLIP suffers severely from negative starvation at small batch sizes, whereas SigLIP remains robust.
• Optimal Operating Point: SigLIP reached peak zero-shot accuracy at $B = 32,768$. Softmax CLIP required scaling batch size to $B = 98,304$ to match its peak, requiring three times the computational and memory overhead.
• Diminishing Returns at Extreme Scale: Beyond $B = 100,000$ (e.g., $B = 307,200$), both losses experience slight degradation due to label noise and false negatives in web-scraped corpora, but SigLIP maintains higher training stability throughout.

Adoption in Modern Vision-Language Models

Because of its superior representation density and memory efficiency, SigLIP has replaced standard OpenAI CLIP as the default vision encoder in modern multimodal architectures:

1. PaliGemma and PaliGemma 2 (Google): Uses a SigLIP-So400M (Shape-optimized 400M parameter ViT) paired with Gemma language decoders for dense image captioning, visual question answering, and object detection bounding box prediction.
2. LLaVA-NeXT and LLaVA-OneVision: Swapped original CLIP-ViT-L/14 encoders for SigLIP-400M, citing superior fine-grained OCR, document comprehension, and spatial localization capabilities.
3. DeepSeek-VL and Pixtral: Utilize SigLIP backbones to handle high-resolution multi-tile visual token inputs without memory exhaustion during pre-training.
4. SigLIP 2 (2025 Extension): Extends the core sigmoid objective with masked autoencoding (Decoder loss), self-distillation, and native dynamic resolution support, delivering state-of-the-art multilingual zero-shot retrieval and dense feature representation.

Sources

• Zhai, X., Mustafa, B., Kolesnikov, A., & Beyer, L. (2023). Sigmoid Loss for Language Image Pre-Training. IEEE/CVF International Conference on Computer Vision (ICCV), arXiv:2303.15343.
• Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning Transferable Visual Models From Natural Language Supervision. International Conference on Machine Learning (ICML), arXiv:2103.00020.
• Lee, J., et al. (2024). Analysis of Using Sigmoid Loss for Contrastive Learning. AISTATS 2024, PMLR 238.
• Tschannen, M., Mustafa, B., Wang, X., et al. (2025). SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features. Google DeepMind, arXiv:2502.14786.
• Beyer, L., et al. (2024). PaliGemma: A versatile 3B VLM for transfer. Google DeepMind, arXiv:2407.07726.