Reading a crash log when you have no symbols

A crash log gives you the place where the machine stopped. It rarely gives you the place where the bug started. A bad pointer can travel through ten functions before something finally reads from it and faults. If you start at the fault and stare at it, you learn how the system died. You do not learn why.

I read these logs in a fixed order. The order matters because each step narrows the next one.

Start with the fault itself

The fault type tells you what the CPU tried to do. A read from an address that is not mapped is a different problem from a write to a read only page, and both are different from executing an address that holds data.

Write down three things before you look at anything else. The faulting address. Whether it was a read, a write, or an execute. Whether the page was missing or present with the wrong permissions.

Then look at the address as a number. Zero or close to zero usually means a null pointer plus a field offset. A small offset from zero is useful. It often matches the offset of a struct field, and that tells you which access failed even without symbols. An address full of a repeated byte pattern usually means the pointer came from cleared, freed, or uninitialized memory. An address that looks like ASCII is often a string read as a pointer by mistake.

Read the registers as a story

The instruction pointer is where execution stopped. The stack pointer tells you whether the stack itself is still believable. If the stack pointer points outside any stack you know, the rest of the log may be built on a broken stack, and the return addresses on it may be garbage.

Check the register that held the bad pointer. Then find where that register was loaded from. In the disassembly around the fault, walk backwards one load at a time. Each load has a source. A register, a stack slot, or a memory address. Keep moving the question back. Where did this value come from. Stop when you reach a value you can explain.

This is slow work. It is also the work. Guessing at this stage costs days, because a wrong guess sends you to a part of the system that is not broken.

Treat the stack as a hint, not as proof

Without symbols, a stack trace is a list of return addresses. Those addresses still help. They show the order of calls, and the distance between two return addresses in the same function gives you a rough sense of where you are inside it.

But a corrupted stack produces a clean looking trace that is wrong. Before you trust it, check that each return address points just after a call instruction in the disassembly. If it does not, treat that frame as suspect and say so in your notes.

Keep a written trace

I keep a plain text file next to every hard log. One line per fact. Address, source, and what I checked. No theories in the fact list. Theories go in a separate list, and each one gets the check that would kill it or confirm it.

This sounds basic until the third day of the same bug. On day three you will not remember which addresses you already ruled out on day one. The file remembers.

Change one thing, then read the new log

When you think you found the cause, change the smallest thing that tests it and run again. Compare the two logs field by field. If the fault moved, your change touched the real path. If nothing changed, your theory was wrong or your change never ran. Both answers are useful. A fault that stays identical after a change tells you to stop and recheck what actually executed.

The rule I keep

The log is the ground truth. My mental model of the system is not. When the two disagree, the model loses, every time. Most long debugging sessions I have had came from defending a model the log had already disproved.

Start at the fault. Work backwards one load at a time. Write down only what the log shows. The bug is usually ordinary, and it is usually upstream of the crash.