When you use a library as massive as pandas, you assume it’s bulletproof. But every few weeks, someone finds a weird edge case that brings the whole thing crashing down.

A few weeks ago, I was looking through the pandas issue tracker and stumbled upon an open bug: to_json(date_format="iso") was causing a hard segfault (a critical memory crash) under specific conditions.

I’m a 17-year-old high school student, but I love debugging C-level memory issues. Here is the story of how I traced the bug, fixed it, and got it merged into pandas 3.2 (PR #69485).

The Bug: The NULL Pointer Dereference

The issue was related to how pandas handles ISO date formatting when exporting to JSON. The crash happened because of a classic C problem: a NULL pointer dereference.

In the C extensions, the JSON encoder uses callbacks like NpyDateTimeToIsoCallback and PyDateTimeToIsoCallback. These callbacks call helper functions (int64ToIso, PyDateTimeToIso) that convert the date.

The problem? If the conversion failed (for example, if a memory allocation failed, or if a tzinfo.utcoffset() raised an exception), the helper function returned NULL.

The callbacks forwarded this NULL string to the ultrajson encoder without setting the errorMsg. Because the encoder only checked errorMsg, it didn't know an error had occurred, and it tried to dereference the NULL string. Boom. Segfault.

The Fix

The fix required changes in two specific places:

  1. Setting errorMsg: I modified the three callbacks in ultrajsonenc.c and objToJSON.c to explicitly set enc->errorMsg whenever the underlying conversion returns NULL. This ensures the pending exception propagates cleanly instead of being ignored.
  2. Fixing apply_tzinfo_offset: In pd_datetime.c, the apply_tzinfo_offset function was returning success even when utcoffset() raised an exception. This caused a secondary SystemError ("returned a result with an exception set"). I modified it to return -1 when an exception is raised so it propagates correctly.

I also added a regression test using a custom tzinfo class whose utcoffset raises a RuntimeError, which reliably triggered the segfault on the main branch.

The Review Process (and AI Disclosure)

Getting a C-level fix merged into pandas isn't just about writing the code; it's about surviving the review.

The PR went through 4 rounds of review with jbrockmendel. He caught redundant defensive checks, asked for cleaner control flow, and pointed out that my whatsnew entry was in the wrong section.

I also want to be transparent: I used Claude to help with root-cause analysis and build verification. I clearly disclosed this in the PR. The maintainers didn't care that I used AI; they cared that the code was correct, the tests passed, and the fix was sound.

What I Learned

  1. C-level bugs are terrifying but logical. Tracing a NULL pointer across a Python/C boundary is hard, but it teaches you how memory actually works.
  2. Open-source maintainers are strict, but fair. They will reject your code 3 times to make it 10x better. Don't take it personally.
  3. You don't need to be a senior engineer to contribute. You just need to be willing to read the source code, write a test, and push through the review cycle.

The fix is now merged into the pandas 3.2 milestone. You can check out the PR here.

If you're a student thinking about contributing to open source, just pick an open issue and start digging. You might break things, but you'll learn more than any tutorial could teach you.