GLM-5.3-Flash combines linear and sparse attention, a mixture-of-experts design, and manifold constrained hyperconnections. A scaled-down version with 25 million parameters can be built in Python and trained on a standard CPU.

We just published a course on the freeCodeCamp.org YouTube channel that will teach you how to build and train a multimodal language model from scratch, covering both pre-training and post-training. The course is from @vukrosic, and all of the code, slides, and experiments are in the GitHub repository linked in the video description.

The model uses a 260-token byte-level vocabulary, a 192-token context, and a small vision encoder that turns a 32 by 32 image into 16 tokens.

In this course, you will learn:

  • Tokenization & Embeddings
    Why a small model needs a small byte-level vocabulary and how weight tying shares one matrix between the embeddings and the output head.
  • Linear & Sparse Attention
    How linear attention keeps a fixed-size memory of recent tokens while every fourth layer uses sparse attention with an indexer to find distant details.
  • Mixture of Experts
    How tokens are routed to a few chosen experts plus a shared expert that learns information common to every token.
  • Adding Vision
    How images are split into patches with 2D RoPE and merged into image tokens so the model can recognize digits.
  • Pre-Training Experiments
    How data diversity, interleaving, and curriculums affect learning, including how interleaving helps prevent catastrophic forgetting.
  • Reinforcement Learning
    How to design reward functions, compute advantages as in GRPO, update only the final layers, and test group size, temperature, and seeds.

After reinforcement learning, the model goes from solving 0 of 24 Python tasks to 16 of 24.

Watch the full course on the freeCodeCamp.org YouTube channel (44-minute watch).