Subscribe
educational
5 min read

What Is a Tensor? The Word AI Stole From 19th-Century Physics

In machine learning, a tensor is just an n-dimensional array of numbers with a shape, a dtype and a device. The word is 180 years older than the field, and the frameworks add autograd, immutability and GPU placement to a plain grid of numbers.

Olivetti-style poster reading WHAT IS A TENSOR next to a dot, a row of squares, a grid of squares and a cube
Olivetti-style poster reading WHAT IS A TENSOR next to a dot, a row of squares, a grid of squares and a cube

Every PyTorch tutorial, every model card and every Hugging Face error message talks about tensors. The word arrives as if everyone knows it, and in machine learning the honest definition is short: a tensor is an array of numbers. That answer is enough to write code. It is not enough to explain why the field uses a word from 19th-century physics instead of the word "array," or what the frameworks actually add to a plain array.

An array that can have more than two directions

A single number is a scalar. A list of numbers is a vector. A grid of numbers with rows and columns is a matrix. A tensor is what you get when you keep going. Goodfellow, Bengio and Courville's Deep Learning textbook puts it plainly: an array of numbers arranged on a regular grid with a variable number of axes is a tensor. The number of axes is called the rank. A scalar is rank 0, a vector rank 1, a matrix rank 2, and a stack of matrices is rank 3.

The TensorFlow documentation works through the same ladder with examples: a rank-3 tensor with shape [3, 2, 5] can be thought of as three matrices of two rows and five columns, one for each entry in the first axis. Every element of a tensor has the same numeric type, which frameworks call the dtype, and the shape, the dtype and the values together are the entire content of the object.

A dot, a row of squares, a grid of squares and a cube, showing rank 0 through rank 3

There is nothing mystical in that structure. Programmers had n-dimensional arrays long before deep learning popularized the word tensor. The difference is not the arrangement of numbers. The difference is what the frameworks hang off them.

Shape, dtype, device

A tensor in a machine learning library carries three pieces of metadata that decide what you can do with it. The shape tells you how many elements sit along each axis. The dtype tells you the numeric type, for example float32 or bfloat16. The device tells you where the bytes live, in your CPU's RAM or on a GPU. PyTorch's tensor constructor accepts all three as arguments, and its documentation defines the device object as the type and index of the accelerator, with cpu, cuda, mps, xpu and xla among the options. Tensors start on the CPU and are moved to the GPU explicitly, which is why every PyTorch training script contains a call to .to(device).

Inside a language model, tensors are the entire inventory. An embedding table is a rank-2 tensor, one row per vocabulary entry. The activations that flow through a transformer layer during a forward pass are rank-3: batch, token position, hidden dimension. The attention weights are rank-4. Hugging Face's documentation for T5 lists the attention weights of each layer as a tensor of shape (batch_size, num_heads, sequence_length, sequence_length). When people say a model has billions of parameters, they are describing the total number of elements across all its weight tensors, which is exactly what a checkpoint file stores. The dtypes are what quantization and mixed-precision training change, and the grid of numbers is what GPUs exist to multiply.

Why not just say array

Frameworks turned the word into the name of a class of objects because those objects do work a NumPy array does not. A PyTorch tensor can be flagged with requires_grad, which makes autograd record every operation performed on it so gradients can be computed later by backward(). A tensor can move between devices, so the same code runs on a laptop CPU and a GPU cluster. In TensorFlow, tensors are the values flowing between the nodes of a dataflow graph, and the library's name comes from exactly that picture: operations that neural networks express as tensor manipulations, arranged in a graph where data flows from inputs to outputs. Even the hardware borrowed the word: Nvidia names its data-center GPUs after the object, the A100 Tensor Core GPU being the best known, its tensor cores doing the matrix multiply-accumulate work that dominates training.

Isometric cubes flowing through a graph of nodes with arrows and a branch

The arrays themselves are usually immutable in the frameworks' own sense. TensorFlow's docs note that you can never change the contents of a tensor in place; you can only build a new one. What persists is the computation around the tensor, not the tensor.

A word 180 years older than the field

Machine learning did not coin the term. William Rowan Hamilton introduced the word tensor in 1846, and it meant something quite different from today's usage: for Hamilton, the tensor of a quaternion was its length, the square root of the product of the quaternion and its conjugate. The modern meaning arrived through mechanics. Woldemar Voigt began using tensor for the physical quantities describing stress and strain in crystals in 1898, around the time Gregorio Ricci-Curbastro was developing tensor calculus, the arithmetic of these objects. Einstein learned the machinery from Marcel Grossmann, used it to write general relativity in 1915, and made the word famous in physics. The stress-energy tensor and the curvature tensor of spacetime are the ancestors of the vocabulary your training loop uses.

Mathematicians also kept a stricter definition than the one software engineers use. In the formal sense, a tensor is an object whose components follow specific transformation laws when you change coordinate system. The number grid is only part of the story; the same physical quantity written in different coordinates produces different component arrays that are all considered views of one tensor. An array in PyTorch has no transformation law, which is why purists say deep learning appropriated the word loosely. Wikipedia covers the disagreement in a dedicated article, "Tensor (machine learning)," which opens by conceding that in the field the term informally refers to a multidimensional array rather than the mathematical object.

The appropriation has consequences beyond pedantry. When a paper says it "exploits the tensor structure" of a video tensor with axes for frames, height, width and color, it means the grid's geometry is meaningful: neighbors along an axis are neighbors in the world, and the operations respect that layout. That is the part of the old word the new field actually kept.

The short version

In machine learning, a tensor is an n-dimensional array of numbers with a shape, a dtype and a device, plus bookkeeping for gradients and hardware placement. It is the universal container: inputs, weights, activations, gradients and checkpoints are all tensors. The word itself came from Hamilton's quaternions, through Voigt's crystals and Einstein's relativity, and arrived in deep learning carrying a mathematical definition that frameworks mostly ignore. If a page of documentation uses tensor where you would say array, you can read it as array and lose nothing except a small argument with a geometer.

Sources

educationalExplainers