The wrong shortage.

Ask a team that trains models what's holding them back, and most will say compute. They want more GPUs, newer GPUs, a bigger reservation, a cheaper hour. There's a whole industry happy to sell each of those.

None of those runs was waiting on arithmetic.Then look at what actually stopped the last few runs. A job crashed overnight with an out-of-memory error. Someone halved the batch size to get it going again, and the schedule slipped. A model got split across eight GPUs when the arithmetic said four would do. And somewhere in a planning doc, an experiment never got proposed because everyone already knew it wouldn't fit.

None of those runs was waiting on arithmetic. Each one hit the limit of the memory on the GPU, the VRAM. Teams don't all hit it in the same place. Plenty of fine-tuning teams have made the model fit with adapters and lower precision, and they meet the limit later: in long sequences, or in the memory an RL run spends generating its attempts. We think this is the shortage most teams pushing their hardware are living with, and the one most often misdiagnosed, because it tends to show up as a request for more compute.

Where the memory goes

GPU memory the modelgradientsoptimizer stateactivations sequence length → Fig. 1 · The model is often the smallest thing on the GPU. Sketch, not to scale.In our first piece we walked through a model with seven billion parameters. It's about 14 gigabytes on disk and many times that once training starts. It's worth going one level down, because where the memory goes decides which workarounds help.

A training run holds four things. The weights are the model itself. The gradients are one number per weight, saying which way to nudge that weight on this step. The optimizer state is the optimizer's running notes. Adam, the most common optimizer, keeps two statistics for every weight, usually at higher precision than the weights. Many setups also keep a higher-precision master copy of the weights. Those three grow with the size of the model. Together they're most of the reason a model that fits on a GPU to use doesn't fit on it to train.

The fourth is activations. These are the intermediate results from the forward pass, kept so the backward pass can use them to work out the gradients. Activations grow with the model's depth and width. They also grow with how much you feed it at once, which is the batch size and the sequence length. Push the context length up and activation memory grows at least as fast. Activations are the part that moves with every decision a team makes about how to train.

Reinforcement learning adds more. Many RL setups hold more than one model at a time: the model being trained, a frozen reference copy that keeps it from drifting too far, and in some setups a model that scores its answers and another that estimates how well an answer is going. Then there's generating the attempts the model learns from. Every attempt in progress holds its own working memory, and that grows with how long each attempt runs and how many run at once. For teams training agents on long tasks, that can be where the memory runs out first.

So the model is often the smallest thing on the GPU. Most of the memory goes to everything training needs around it.

What teams do about it

memorypaid withAdapterswhat the model learnsSmaller batcheswhat the model learnsCheckpointingtimeShardingGPUsOffloadingtime Fig. 2 · Every workaround pays for memory with something else. Sketch, not to scale.Nobody who trains models is new to this, and there's a well-worn set of workarounds. Every one of them works, and every one of them costs something.

AdaptersMuch of today's fine-tuning freezes the model and trains small adapter layers alongside it, a method called LoRA, often with the frozen model stored in 4 or 8 bits. It's a big part of why a team can fine-tune a large model on a single GPU at all. The cost is that only the adapters learn. Published comparisons find LoRA learns less than full fine-tuning in standard settings, though it also forgets less of what the model already knew. For many runs that trade is fine, especially when what's being measured is the difference between two versions of a model.

Smaller batchesSmaller batches or shorter sequences cut activation memory directly, and the cost lands on the model. Shorter sequences mean it sees less context. Smaller batches can make training noisier. Teams can win the batch size back by accumulating gradients over several small steps, but that costs time.

CheckpointingGradient checkpointing keeps only some of the activations and recalculates the rest during the backward pass. It saves a lot of memory and adds compute to every step.

ShardingSharding splits the weights, gradients and optimizer state across several GPUs, so each holds a slice. It's a standard way to train large models today. It costs more GPUs, and time spent passing data between them.

OffloadingOffloading moves part of the training state out to ordinary CPU memory, which is plentiful. The link back to the GPU is much slower than the GPU's own memory, and moving data across it adds time.

Most large runs use several of these at once. Tuning the combination is a skill in its own right, and it's time spent on what fits instead of on the data and the results.

We don't think any of these are mistakes. They're the right answers to the constraint as it stands today. But each one pays for memory with something else, whether that's time, GPUs or what the model learns, and teams pay it again on every run.

Why buying more doesn't close it

The obvious fix is newer GPUs with more memory, and every generation does bring more. The trouble is the rate. In the transformer's first five years, the largest models grew roughly 410 times every two years, while the memory on a single GPU grew about two times.

There are good reasons memory lags. The fast memory on a modern GPU is built by stacking thin memory chips and packaging them right next to the processor. Adding more of it runs into heat, into limited space around the chip, and into manufacturing yield. Adding arithmetic is easier. The people working on memory aren't slow. It's just a harder problem.

When a team adds GPUs to make a run fit, it's buying compute to get memory.That leaves the industry in an odd spot. It keeps buying more of the resource that grows fastest, compute, to relieve a shortage of the resource that grows slowest, memory. When a team adds GPUs to make a run fit, it's buying compute to get memory.

And many teams can't simply buy more. They've leased a fixed cluster for months at a time, so the real question is how many useful runs it gets through. Every run that has to be cut down or restarted to fit is one it didn't.

Why it matters more every year

Training used to be an event: one big run, a finished model, then months of using it. For a growing number of teams it's something they do most weeks: fine-tunes for new tasks, preference training, reinforcement learning on agents that keep changing. Every one of those runs meets the same memory limit and works around it the same way. What used to be a one-time planning problem is now a standing constraint the team designs around, over and over.

The lever that's left

If a team can't reliably get more bytes, the thing left to change is how much training each byte holds. Every workaround above is an answer to that question, and each one answers it by giving something up.

We think there's a better answer in how training state is represented. Weights, gradients and optimizer notes aren't random. A method that finds the pattern in them and stores the pattern should let the same GPU hold a bigger training run, while giving up less than one that shrinks everything by the same amount. Compression has a cost too, and we'll keep saying so. Our methods are designed to keep that cost low and to let a team choose it and measure it. The math we start from comes from physicists who hit the same wall decades ago, which we wrote about in It starts with the math.

If you've halved a batch, sharded a model or shelved an experiment to make a run fit, tell us what you train. Perhaps the run fits after all.