Almost every gain in training efficiency over the last few years came from making the machine better. Faster chips. Tighter kernels. Smarter parallelism. Lower precision. Attention that reads and writes memory more carefully. Each of these is a real advance, and stacked together they have made training dramatically faster.
They also share a quiet assumption. Every one of them makes the same computation run faster. The math the machine is solving does not change. We keep building better engines for a route nobody thinks to redraw.
Two axes, not one
It helps to separate two different questions. The first is how fast can we run this computation. That is the axis the whole industry competes on, and it is crowded with brilliant work. The second is a different question entirely. Is this the computation we should be running at all.
Training a model is, underneath the frameworks, one enormous mathematical problem solved over and over, millions of times. The shape of that problem is not handed down by nature. It is a choice, and choices can be made better. Re posing the problem so that the same answer takes less work is a lever that sits underneath the chips and the kernels, not beside them.
Why it stacks instead of competes
This is the part people find surprising. Because the work happens at the level of the problem rather than the hardware, it does not compete with the tools a lab already runs. Mixed precision still helps. FlashAttention still helps. Sharded training still helps. A better-shaped problem runs through all of them and comes out the other side cheaper. You do not rip anything out.
We are not going to describe the method here. What matters for this post is the framing, because the framing is what almost everyone skips. The room full of people optimizing the engine is enormous. The room asking whether the route is right is nearly empty. That is where we chose to work.