FlashAttention: Why Tiling Attention Makes Transformers Fast Without Changing the Result

Transformers did not get slow because attention needs too many multiplications. They got slow because attention moves too much data. FlashAttention is a lesson in that distinction. It does not change what attention computes. It changes where the intermediate results live, how often they are read and written, and in what order the work happens. The result is the same attention output, computed with far less traffic to slow memory, and therefore much faster in practice, especially for long sequences.

This article explains the memory wall behind standard attention, the online softmax trick that makes tiling possible, the forward and backward passes of FlashAttention, and the limits that remain. If you have ever watched a model run out of memory at a modest sequence length, or wondered why a theoretically quadratic operation can still be made practical, this is the mechanism to understand.

The problem is memory, not arithmetic

Standard scaled dot-product attention starts with three matrices for one head: queries $Q$ , keys $K$ , and values $V$ . For a sequence of length $N$ and head dimension $d$ , each is $N \times d$ . The output is:

Attention (Q, K, V) = softmax (\frac{Q K ^{T}}{d}) V

The score matrix $S = Q K^{T} / d$ is $N \times N$ . That is the first problem. For $N = 8192$ , one head already produces about 67 million scores. Store that in 16-bit precision and you need roughly 134 megabytes for a single head, for a single layer, for a single sequence. Multiply by heads, layers, and batch size, and the score matrix dominates memory long before the weights do.

The second problem is worse than the storage itself: the traffic. A standard implementation typically materialises $S$ in high-bandwidth memory (HBM), reads it back to apply the softmax, writes the probability matrix $P$ back to HBM, and then reads $P$ again to multiply by $V$ . Every one of those passes crosses the slowest link in the system.

Modern GPUs have a memory hierarchy, and the gap between its levels is the whole story. HBM is large but relatively slow. On-chip SRAM is tiny but very fast, often an order of magnitude higher in bandwidth. A standard attention implementation is therefore memory-bound: the arithmetic units spend much of their time waiting for data to arrive from HBM. Counting floating point operations misses this completely. Two algorithms with the same operation count can run at very different speeds if one of them reads and writes $N^{2}$ values to HBM and the other does not.

FlashAttention, introduced by Dao and co-authors, attacks exactly this gap. Its goal is to never materialise the full $N \times N$ matrix in HBM at all.

Why softmax seems to block tiling

Tiling is the classic fix for a memory-bound operation: break the matrices into blocks that fit in fast SRAM, compute on one block at a time, and keep the partial results on chip. Matrix multiplication tiles beautifully. Softmax does not, at first glance, because it is a row-wise operation that needs the whole row before it can produce any output.

For one row of scores $x = (x_{1}, \dots, x_{N})$ , the softmax is:

softmax (x)_{i} = \frac{e ^{x_{i} - m}}{\sum _{j} e ^{x_{j} - m}}, m = j max x_{j}

The subtraction of the maximum $m$ is there for numerical stability: it keeps the exponentials at or below 1 and prevents overflow. But it creates a dependency. You cannot compute the denominator, the sum of exponentials, until you have seen every element of the row and found its maximum. If you process the row in blocks, the first block cannot be finished until the last block has been seen. That seems to force either storing the whole row or making two passes over it.

Online softmax removes that obstacle. The trick is to keep running statistics and to correct earlier partial results when new information arrives, rather than waiting for the complete row.

Online softmax: correct as you go

Suppose you process a row in blocks. After the first block, you have a running maximum $m_{1}$ and a running sum of exponentials $l_{1}$ :

l_{1} = j \in block 1 \sum e^{x_{j} - m_{1}}

Now a second block arrives with a larger maximum, so the new running maximum becomes $m_{2} > m_{1}$ . Every exponential computed against $m_{1}$ is now off by a common factor, because:

e^{x_{j} - m_{2}} = e^{x_{j} - m_{1}} \cdot e^{m_{1} - m_{2}}

So the old running sum can be rescaled, not recomputed:

l_{2} = e^{m_{1} - m_{2}} \cdot l_{1} + j \in block 2 \sum e^{x_{j} - m_{2}}

The same rescaling applies to the partial output. Attention output for a row is a weighted sum of value vectors, where the weights are the softmax probabilities. If you have accumulated a partial output $O_{1}$ from the first block, normalised by the old sum $l_{1}$ , you can rescale it by the same factor and add the contribution of the new block. In general, after processing blocks up to block $t$ , with running maximum $m_{t}$ and running sum $l_{t}$ , the partial output is:

O_{t} = \frac{1}{l _{t}} s \leq t \sum e^{m_{s - 1} - m_{t}} \cdot (block s contribution)

In practice the implementation keeps the unnormalised accumulator and divides once at the end, which avoids a division per block. The key point is conceptual: softmax is no longer a barrier to tiling. Each block can be processed, folded into running statistics, and discarded. No block needs to be stored after its contribution has been absorbed.

This is the same idea as computing a running mean or a running maximum in a single streaming pass. The only extra ingredient is the rescaling factor, which is a single scalar per row, so correcting all earlier work costs almost nothing.

The tiled forward pass

With online softmax in hand, the forward pass of FlashAttention looks like this for one query block:

Load a block of queries $Q_{i}$ into SRAM, together with running statistics $m_{i}$ and $l_{i}$ and the partial output $O_{i}$ .
Loop over key and value blocks $(K_{j}, V_{j})$ , one pair at a time:
- Load $K_{j}$ and $V_{j}$ into SRAM.
- Compute the block scores $S_{ij} = Q_{i} K_{j T} / d$ on chip.
- Find the block maximum, update the running maximum, and rescale $l_{i}$ and $O_{i}$ .
- Compute the block softmax weights and accumulate $P_{ij} V_{j}$ into $O Source：https://dev.to/umeshtharukaofficial/flashattention-why-tiling-attention-makes-transformers-fast-without-changing-the-result-1gp1 My favorites$