llms.blog
Vision Transformers2 articles

Vision Transformers

Articles

  • Self-Distillation with No Labels (DINO): How Momentum Teachers, Centering, and Sharpening Emerge Semantic Attention in Vision Transformers

    Self-Distillation with No Labels (DINO): How Momentum Teachers, Centering, and Sharpening Emerge Semantic Attention in Vision Transformers When the Vision Transformer (ViT) was introduced by Dosovitskiy et al. in 2020, standard wisdom suggested that transformers required massive supervised corpora (such as JFT-300M) to overcome their lack of convolutional inductive biases. Unlike Convolutional Neural Networks (CNNs), which bake translation equivariance and local receptive fields directly into t

    1 min

  • Masked Autoencoders: How Asymmetric Encoders, High Masking Ratios, and Pixel Reconstruction Scaled Vision Transformers

    Masked Autoencoders: How Asymmetric Encoders, High Masking Ratios, and Pixel Reconstruction Scaled Vision Transformers Self-supervised pre-training transformed natural language processing through masked language modeling, popularized by BERT (Devlin et al., 2018). By hiding a subset of input tokens and training a bidirectional Transformer to predict the missing words from context, models learned rich, generalizable linguistic representations without manual annotations. Adapting this masked pre

    1 min