Every large language model you have ever used was trained on this and serves from it. It is not a chip. It is not a framework. It is a software platform from NVIDIA called CUDA, and it is the reason a Python string, "cuda", can move a model onto a graphics card and make it go fast. This explainer covers what CUDA actually is, how a line of PyTorch travels through it down to silicon, and why the entire industry's switching costs live at this layer.
What CUDA is
CUDA, which originally stood for Compute Unified Device Architecture, is NVIDIA's proprietary parallel computing platform and programming interface for its own GPUs. It lets software use a graphics processor for general-purpose computation rather than only for drawing pixels. NVIDIA began building it in 2004, after hiring Ian Buck, whose Brook research language at Stanford demonstrated that GPUs could run general computation; the toolkit was officially released on February 15, 2007, for Windows and Linux.
Two details in that definition matter. First, CUDA is proprietary: it is an NVIDIA product, licensed by NVIDIA, and it runs on NVIDIA hardware. Second, it comes in two layers, a low-level Driver API and a higher-level Runtime API, so it is a full platform rather than a single library. From the start it worked on ordinary GeForce cards, not just expensive datacenter parts, which turned out to be the strategic point.
How a kernel reaches the chip
CUDA extends C++ with a few keywords that let a developer write a kernel, a function that runs on the GPU instead of the CPU. Kernels are written for thousands of parallel threads, each handling a small slice of a big matrix operation. The rest is compilation plumbing, and it explains most of what people mean when they talk about CUDA lock-in.
The compiler, nvcc, splits your code in two. Ordinary C++ for the CPU goes to your system compiler. GPU code is compiled to PTX, an intermediate instruction set that works something like LLVM IR or Java bytecode. PTX is then assembled into SASS, the native machine code for one specific GPU architecture, identified by numbers like sm_80. Binaries can embed PTX so the driver re-compiles it for whatever GPU a machine actually has, which is why a CUDA program usually runs on a newer card than it was built for, but a SASS binary compiled only for an old architecture has to be rebuilt when the fleet changes.
That rebuild requirement is policy, not accident. The CUDA 13 toolkit, released in 2025, removed offline compilation and library support for Maxwell, Pascal and Volta, GPU families from 2012 to 2017. Code targeting them must be built with the older 12.x toolchain or modernized.
The stack you actually touch
Almost nobody writing AI applications programs CUDA directly today. CUDA is the floor of a stack. One layer up sit NVIDIA's math libraries: cuBLAS handles the matrix multiplications that dominate neural network compute, and cuDNN provides tuned implementations of deep learning primitives such as convolution, attention, matmul, pooling and normalization. NCCL moves gradients between GPUs during distributed training. TensorRT and TensorRT-LLM compile trained models for fast inference serving.
Above that sits the framework layer everyone actually works in: PyTorch, TensorFlow, JAX, and inference engines like vLLM and SGLang. When you call model.to("cuda") in PyTorch, the framework dispatches your operations into cuDNN and cuBLAS kernels, which run on the CUDA runtime, which drives the NVIDIA kernel driver, which talks to the hardware. The dependency chain is the whole story: your code depends on the framework, the framework on the libraries, the libraries on CUDA.

The bet that took five years to pay off
CUDA launched into indifference. NVIDIA itself calls the early market a "zero-billion-dollar market" that the company entered on conviction. The company put CUDA on every GeForce card it sold, wrote textbooks, taught university classes and gave away development kits, betting that a cheap parallel supercomputer in every lab would create demand later. Its own description of the strategy is a virtuous cycle: the installed base attracts developers, developers attract applications, applications widen the base.
The payoff has a specific date. In 2012, the Toronto team of Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton won the ImageNet recognition competition with a deep convolutional network, later called AlexNet, that beat the runner-up's top-5 error by nearly 11 points, 15.3 percent against 26.2 percent. The network trained for five to six days on two GeForce GTX 580 gaming cards with 3 GB of memory each, high-end consumer hardware priced around $500, not datacenter accelerators. Every researcher who read that paper went out and bought a GeForce card and installed the CUDA toolkit. The generation of ML engineers who learned GPU programming in university classes learned it on NVIDIA hardware, and that muscle memory is what the company now monetizes. By the time of its own published corporate history, NVIDIA counted more than 5 million developers and 40,000 companies building on the platform.
Why the moat is software, not silicon
Competitors had the hardware argument first and lost it repeatedly. AMD launched its open ROCm platform in 2016, nine years after CUDA; Intel launched oneAPI in 2020. Late-2026 vendor benchmarks show the gap has narrowed where workloads stick to mainstream frameworks: one cloud provider's testing puts ROCm at 90 to 95 percent of H100 throughput for standard LLM inference running PyTorch, vLLM or SGLang on AMD MI300X and MI355X parts.
The gap that remains is exactly where CUDA-specific code lives. The same comparisons find that TensorRT-LLM and FlashAttention 3 have no ROCm equivalents, and hand-written CUDA kernels and inline PTX require manual porting. Standard models run anywhere; the tuned, unusual, fast-at-the-margin parts of a stack are CUDA-anchored. Switching providers stops being a hardware purchase and becomes an engineering project, which is the moat.

The ladders over the wall
Two developments in the last few years aim at this lock-in, and both work above CUDA rather than against it head-on.
The first is Triton, an open-source Python-based language for writing GPU kernels that OpenAI released in July 2021. Instead of hand-tuning CUDA per architecture, a developer expresses a computation in tile-level Python and a compiler generates the hardware code. PyTorch 2.0, released in March 2023, made Triton the default GPU code generator inside torch.compile for both NVIDIA and AMD hardware, explicitly to hide low-level hardware details. Note what that does and does not accomplish: most Triton code on NVIDIA GPUs still lands on the CUDA driver stack underneath. It lowers dependence on hand-written CUDA; it does not remove NVIDIA hardware from the picture.
The second is Google. Reuters reported in December 2025 that Google had stood up an internal initiative called TorchTPU to make its TPUs run PyTorch natively, because customers building on PyTorch viewed the software stack, not the silicon, as the bottleneck for adopting TPU chips. TPUs were long optimized for Google's own JAX framework; making CUDA-free PyTorch feel native is an attempt to delete the rewrite cost that protects NVIDIA's position. Google was reported to be considering open-sourcing parts of the software.
Why it matters for anything LLM-shaped
Every price per million tokens, every GPU availability story, every claim about open models running on consumer hardware is downstream of this layer. Model pricing sheets are, mechanically, CUDA pricing sheets. When you read that a lab trained a frontier model on 100,000 GPUs, read that as 100,000 units running a software platform NVIDIA controlled end to end, with libraries it tunes and a compiler cadence it sets. The hardware press coverage rotates through new chips each year; the programming model underneath has been the same since 2007, and that continuity is the actual product.
The practical lesson for teams: the portable part of your AI stack is the part that never touches a kernel. Framework-level code moves. Custom CUDA does not. Every vendor dependency you accept at the CUDA layer is one you will re-pay in engineering time the day you want a cheaper accelerator.
Sources
- CUDA - Wikipedia
- NVIDIA, Our Story (corporate history PDF)
- CUDA Toolkit: Compilers, Libraries, and the Host-Device Model - alonge.dev
- CUDA Toolkit 13.1 Release Notes - NVIDIA
- NVIDIA cuDNN - NVIDIA Developer
- AlexNet - Wikipedia
- NVIDIA Software Ecosystem for AI - American Compute
- ROCm vs CUDA 2026: Benchmarks, Gaps and Real Cost - Spheron
- ROCm vs CUDA: GPU Computing Comparison - Thunder Compute
- Introducing Triton: Open-source GPU programming for neural networks - OpenAI
- PyTorch 2.0 release announcement - PyTorch
- Google works to erode Nvidia's software advantage with Meta's help (Reuters reporting) - Virginia Business
