Why I started looking into causal masking

I've been working on a personal learning project called Building LLM Technologies, where I explore individual components of language models using small Python notebooks.

The idea is to understand one mechanism at a time, without having to build or train an entire language model.

One of the first things I looked at was causal masking.

It's a relatively simple concept, but an important one for understanding how autoregressive language models work.

What's a causal mask?

When a model processes a sequence of tokens, attention allows each position to use information from other positions.

But in a causal language model, a position shouldn't be able to look at future tokens when predicting what comes next.

For example, consider:

["I", "love", "Python"]

The first token can only attend to itself. The second can attend to the first two tokens. The third can attend to all three.

We can represent this with a simple matrix.

tokens = ["I", "love", "Python"] mask = [ [1, 0, 0], [1, 1, 0], [1, 1, 1] ] for row in mask: print(row)

Output:

[1, 0, 0] [1, 1, 0] [1, 1, 1]

Here, 1 means attention is allowed, while 0 means that position is blocked.

The triangular structure ensures that each position can only access itself and earlier positions.

Applying the mask to attention scores

A Transformer normally calculates attention scores from Query and Key vectors.

To keep this example simple, I used predefined scores:

scores = [ [2, 2, 0], [2, 4, 2], [0, 2, 2] ] masked_scores = [] for i, row in enumerate(scores): new_row = [] for j, score in enumerate(row): if mask[i][j] == 1: new_row.append(score) else: new_row.append(-999) masked_scores.append(new_row) for row in masked_scores: print(row)

Output:

[2, -999, -999] [2, 4, -999] [0, 2, 2]

The scores for future positions have been replaced with a very large negative value.

I used -999 to keep the example straightforward. Real implementations generally use negative infinity or an appropriate masking value.

What happens after softmax?

Attention scores are converted into probabilities using softmax.

Here's the minimal implementation from my notebook:

import math def softmax(row): exps = [math.exp(x) for x in row] total = sum(exps) return [x / total for x in exps] for row in masked_scores: print(softmax(row))

Output:

[1.0, 0.0, 0.0] [0.11920292202211756, 0.8807970779778824, 0.0] [0.06337893833303762, 0.4683105308334812, 0.4683105308334812]

And that's the interesting part.

The first position assigns all its attention to itself. The second distributes attention across the first two positions. The third can attend to all three.

With the large negative masking value used here, future positions receive attention weights that are effectively zero after softmax.

This is how masking prevents information from later positions from influencing earlier ones.

What this example doesn't cover

This is intentionally a small educational example.

The attention scores are predefined rather than calculated from learned Query and Key projections. There's no training process or complete Transformer implementation.

The softmax function is also simplified and isn't intended as a numerically stable production implementation.

I wanted to isolate the masking mechanism and make its effect visible through a few small matrices.

Why I found this useful

What I like about studying LLM components this way is being able to inspect every intermediate result.

Instead of starting with a complete Transformer and trying to understand everything at once, I can focus on one mechanism, see what changes, and then move on to the next.

Causal masking was the first of ten small notebooks I created following this approach.

I've made the project available on GitHub and Kaggle in case anyone else finds it useful for learning.

Have you tried learning Transformer internals by implementing individual components from scratch?

Writing note: I used AI assistance to translate, organize, and polish this article in English. The project, code examples, and technical content are my own.