Tensor Shapes, Dtypes and Devices Explained

Every PyTorch tensor has three attributes you should check first when something breaks: shape (its dimensions), dtype (the type of each number) and device (where the data is stored, such as CPU or GPU). Most beginner errors come from two tensors disagreeing on one of these three. This post shows how to read them and convert between values.

Shape

The shape is a torch.Size, which behaves like a tuple of integers.

import torch

x = torch.rand(8, 3, 32, 32)    # a batch of 8 RGB images, 32x32

print(x.shape)       # torch.Size([8, 3, 32, 32])
print(x.ndim)        # 4
print(x.size(0))     # 8  (size of dimension 0)
print(x.numel())     # 24576 (total number of elements)

By convention image batches in PyTorch use the layout (batch, channels, height, width). Text batches are often (batch, sequence_length) of token ids, and feature batches (batch, features). The first dimension is nearly always the batch.

Dtype

The dtype controls precision, memory use and which operations are allowed. The ones you will meet most often:

• torch.float32: the default for model weights and activations.
• torch.float16 and torch.bfloat16: half precision, used in mixed-precision training to save memory and time.
• torch.float64: double precision, rarely needed in deep learning but common in scientific code.
• torch.int64 (also torch.long): class labels and indices.
• torch.bool: masks and comparison results.

x = torch.tensor([1, 2, 3])
print(x.dtype)                  # torch.int64

f = x.float()                   # to float32
d = x.to(torch.float64)         # to float64
h = f.half()                    # to float16
b = (f > 1.5)                   # bool tensor

print(f.dtype, d.dtype, h.dtype, b.dtype)
print(torch.get_default_dtype())  # torch.float32

Use .to(dtype) for general conversion; .float(), .long() and .bool() are shortcuts. Conversions return a new tensor and leave the original unchanged.

The classic dtype errors

Two errors are so common they are worth recognising on sight. A message like "mat1 and mat2 must have the same dtype" or "expected scalar type Float but found Double" means you mixed precisions, often because a NumPy array (float64) was turned into a tensor and fed to a float32 model. Fix it with .float(). The other is "expected scalar type Long" from a loss such as cross-entropy, which needs integer class labels, so convert labels with .long().

Device

The device says where the tensor's memory lives. Operations between tensors require them to be on the same device.

device = "cuda" if torch.cuda.is_available() else "cpu"

a = torch.rand(2, 2)            # on CPU
b = a.to(device)                # copy to GPU if available
print(a.device, b.device)

c = b.cpu()                     # back to CPU
print(c.device)

# Combining tensors on different devices raises a RuntimeError
# a + b   ->  "Expected all tensors to be on the same device"

Moving data between CPU and GPU is relatively slow, so move batches once per step and keep intermediate results on the device. A model is moved with model.to(device), and its inputs must be moved too.

Changing several attributes at once

.to() accepts a device, a dtype or both, and can copy those from another tensor:

x = torch.tensor([1, 2, 3])
ref = torch.zeros(1, dtype=torch.float16, device=device)

y = x.to(ref)         # matches ref's dtype and device
z = x.to(device, torch.float32)
print(y.dtype, z.dtype)

A debugging habit

When an operation fails, print one line for each tensor involved:

def info(name, t):
    print(f"{name}: shape={tuple(t.shape)} dtype={t.dtype} device={t.device}")

info("inputs", torch.rand(4, 10))
info("labels", torch.tensor([0, 1, 1, 0]))

The output makes mismatches obvious: for example inputs on cuda:0 and labels on cpu, or labels as floats when the loss needs integers.

Why precision matters

Lower precision uses less memory and runs faster on modern GPUs, but it stores fewer digits. A float16 number has about three decimal digits of precision and a small range, so very large or very small values can overflow or underflow. bfloat16 keeps the range of float32 but with even fewer digits, which makes it more forgiving during training. Because of these trade-offs, frameworks keep a master copy of weights in float32 and only use half precision for selected operations. You will see exactly how this works in the mixed-precision post.

Memory use is easy to estimate from these attributes. A tensor of one million float32 values takes about four megabytes, since each value uses four bytes, while the same tensor in float16 takes two. The method t.element_size() gives the bytes per element, and multiplying by t.numel() gives the total. When a model runs out of GPU memory, this simple arithmetic is often the first step in understanding why.

Checklist

• Model weights and inputs: float32 (or half precision under autocast).
• Classification labels: int64.
• Masks: bool.
• Everything in one operation: same device.

Next you will learn how to pick out the parts of a tensor you need with indexing and slicing.