A lot of LLM tutorials start after the hardest learning has already happened: load a pretrained checkpoint, call a high-level API, and fine-tune it. That is useful for shipping products, but it can leave the most interesting parts hidden.
MiniMind takes the opposite route. It is a compact, PyTorch-first project for understanding the full lifecycle of a small language model: tokenizer, data preparation, transformer implementation, pretraining, supervised fine-tuning, evaluation, and checkpoints.
It appeared at #4 in the November 22 GitHub daily trending snapshot and #7 in the weekly Python list. The archive recorded +488 daily stars and +2,879 over the week. This is renewed attention, not a brand-new launch: the repository dates back to August 2024. Its current main model, MiniMind-3, is a 64M-parameter release dated April 1, 2026.
What problem does MiniMind solve?
Frameworks such as Transformers, TRL, and PEFT make model experimentation much faster. Their abstractions are valuable, but a beginner can spend weeks using them without understanding how the underlying training loop works.
MiniMind keeps more of that machinery visible. The project describes its core algorithms as native PyTorch implementations rather than thin wrappers around a high-level training framework. Its main educational model is deliberately small, so the code and training pipeline are easier to inspect than a production-scale foundation model.
The repository also extends beyond one toy transformer. Its documentation covers pretraining, SFT, LoRA, preference optimization, tool use, reinforcement-learning experiments, distillation, and related variants. Treat those as learning paths to inspect individually, not as a promise that every recipe is equally production-ready.
Credit: architecture visual from the MiniMind repository.
Follow the training pipeline
The useful mental model is a sequence of explicit stages:
- Tokenizer: map text to token IDs and back. The README lists a 6,400-token vocabulary for the current mainline.
- Pretraining: teach the model next-token prediction from text records.
- Supervised fine-tuning: adapt the pretrained weights to a conversation format.
- Evaluation: load the resulting checkpoint and test generation.
- Checkpointing: save model, optimizer, and training progress so an interrupted run can resume.
The default quick-reproduction path uses the smaller pretrain_t2t_mini.jsonl and sft_t2t_mini.jsonl datasets. The full dataset path is larger and intended for a more complete reproduction.
Credit: MiniMind's own Web UI image from the repository.
Set it up
The README's reference environment lists Ubuntu 20.04, Python 3.10.16, CUDA 12.2, 128 GB RAM, and eight RTX 3090 GPUs. That is a reference configuration, not a minimum requirement. The project says CPU and Apple MPS paths are possible, but training speed and compatibility will differ substantially.
Start by cloning the repository and installing its Python dependencies:
git clone --depth 1 https://github.com/jingyaogong/minimind cd minimind pip install -r requirements.txt Install a PyTorch build that matches the CUDA runtime and GPU drivers on your machine before starting a serious training run. Then download the two small dataset files from the official ModelScope dataset page and place them in the repository's dataset/ directory.
Run the training stages from the trainer directory:
cd trainer python train_pretrain.py python train_full_sft.py The scripts support checkpoint recovery. If a run is interrupted, the README documents the resume flag:
python train_pretrain.py --from_resume 1 python train_full_sft.py --from_resume 1 For inference, download the published model and run the evaluation script from the repository root:
modelscope download --model gongjy/minimind-3 --local_dir ./minimind-3 python eval_llm.py --load_from ./minimind-3 The project also documents a Hugging Face download path and optional Streamlit demo. Follow the current README for the exact model directory and dependency versions.
A note about the “two hours” claim
The project explains that its two-hour figure refers to one SFT epoch on a single NVIDIA RTX 3090, not the complete pretraining-plus-fine-tuning lifecycle. The reference environment elsewhere in the README lists eight 3090 GPUs. Those statements describe different reproduction setups; do not turn the headline into a guaranteed end-to-end training benchmark.
Your result will depend on the dataset, model dimensions, batch size, sequence length, precision, storage, and GPU. Start with the mini datasets and a small configuration, then scale only after you can reproduce the baseline.
Who should use it?
MiniMind is a strong fit for students who want to understand transformer training, engineers prototyping tokenizer or attention changes, researchers who need an inspectable baseline, and developers who want to learn how pretraining and SFT checkpoints fit together.
It is not a replacement for a large pretrained model when you need high-quality general-purpose reasoning, production throughput, or broad multilingual performance. A small educational model is valuable because it is inspectable, not because it competes with large commercial systems.
MiniMind versus other learning routes
| Project or route | Best use |
|---|---|
| MiniMind | A compact end-to-end PyTorch learning path |
| nanoGPT | A focused, minimal GPT training implementation |
| LitGPT | More configurable training and deployment workflows |
| Hugging Face Transformers | Practical use of existing pretrained models |
| torchtitan | Distributed training research at a much larger scale |
MiniMind is Apache-2.0 licensed. The code license does not automatically grant every dataset the same rights; inspect the source and terms of each data collection before redistributing a trained model or dataset.
Sources and further reading
- MiniMind repository and English README
- Training and inference instructions
- ModelScope dataset
- ModelScope online demo
- Maintainer's video introduction on Bilibili
The project has an exact maintainer video on Bilibili, but I could not verify a suitable exact-project YouTube walkthrough, so I have not substituted an unrelated video.
If you want to understand one part of an LLM deeply, would you start with the tokenizer, the attention implementation, or the training loop?

