Parts 1 to 3 of this series covered what the converter does: 34 rows across four statement layouts, all fields correct, then a GUI, a Windows installer, and the scanned-PDF path through OCR. This post is about a decision underneath all of that — which PDF library to build on — and about the test fixtures, which nearly hid a real problem instead of catching one.

The choice: one runtime dependency

The converter has exactly one runtime dependency: pdfplumber==0.11.10. It was picked over PyMuPDF and pypdf for two concrete reasons. First, its line-based table extraction fits bank statements, which are mostly ruled grids with debit and credit columns. Second, it is pure Python on top of pdfminer.six, which keeps deployment simple — one package, no system binary to install.

PyMuPDF is still in the project, but only on the test side: it generates the test fixtures. It does not take part in conversion. That split matters, because the fixture generator is where the interesting bug lived.

The fixture bug

To test the converter I generate synthetic statements and compare every extracted row field by field against ground truth. The first attempt used reportlab with an embedded NotoSansCJK font. It looked fine on screen. Extracted text, though, came back as \x00 for every Chinese character — the embedded font's encoding made the text unrecoverable.

The second attempt used PyMuPDF with the same font. Chinese characters extracted correctly this time, but something quieter went wrong: the hyphen - was mapped to U+2011 (non-breaking hyphen), and a few Chinese characters landed on compatibility code points. A converter tested only against those fixtures would have learned to handle U+2011 hyphens as if that were normal — or, worse, the fixtures would have passed while real PDFs with plain ASCII hyphens behaved differently.

The fix was to render fixtures in two parts: CJK text with PyMuPDF's built-in china-s font, ASCII with Helvetica. And the converter itself now runs NFKC normalisation over all extracted text before parsing, so a U+2011 hyphen from any PDF generator is folded back to a plain hyphen before it can break a match. The fixture bug surfaced a real class of problem — PDF generators do emit odd code points — and the normalisation step is the answer to it, not to any single fixture.

What the fixtures actually proved

Four digital layouts, 34 rows, every field compared against ground truth:

  • Two-column debit/credit, two pages, 12 rows: 12/12
  • Single amount column with +/- signs, 8 rows: 8/8
  • Reordered columns with MM-DD dates (year taken from the title), 8 rows: 8/8
  • No table at all, plain text lines through the fallback parser, 6 rows: 6/6

The scanned path (6/6 through OCR) was covered in part 3. The point of listing the four layouts here is that they exercise different code paths: table extraction first, text-line parsing as a fallback when there is no table to extract.

Two decisions that fell out of this

First, columns are recognised by header keywords — date, description, debit, credit, balance, amount, and their Chinese equivalents. If the parser meets header wording it has never seen, it skips that table instead of guessing. A missed table is a visible failure; invented numbers are not.

Second, I am not claiming pdfplumber is the best PDF library in general. It was the best fit for this job — ruled tables, one dependency, no system binary — and it passed these tests. A different document shape (say, statements with no ruling lines at all) might change the answer, and the text-line fallback exists precisely because the table path will not cover everything.

What this does not prove

The same honest limits as before, restated because they still apply: no real bank statement has been tested yet — every fixture is synthetic, and banks differ in header wording, column order, summary rows, and multi-page tables. Encrypted PDFs and image PDFs that mix in a text layer are untested. The 34/34 result says the converter handles the four layouts I built; it says nothing about the fifth layout I have not seen.

The converter is open source: https://github.com/alapha888/bank-statement-presale

The presale page with the full status, scope and refund terms: https://alapha888.github.io/bank-statement-presale/