Why my credit risk model uses a 15% threshold

My credit risk model scores 85.17% accuracy. It's the least interesting number it produces.

The model predicts the probability that a loan applicant will default, using their financial, demographic and credit history. It's a HistGradientBoostingClassifier inside a scikit-learn pipeline, and it powers a small Streamlit app where you enter an applicant's details and get a risk estimate back.

Accuracy on a rare event

Defaults are the minority. When most applicants repay, a model can reach high accuracy by leaning towards "will repay" for almost everyone, and still miss the people a lender most needs to catch.

So the numbers I actually look at are these:

ROC-AUC says how well the model ranks risky applicants above safe ones, independent of any cut-off. Recall says what share of real defaulters it flags. Precision says how many of the flagged applicants really were risky.

Moving the threshold

A classifier outputs a probability. Something has to turn that into a decision, and by default that something is 0.5: flag the applicant only if the model is more than 50% sure they'll default.

For this problem, 0.5 is the wrong question. The two mistakes don't cost the same:

• Missing a defaulter means a loan that may never be repaid.
• Flagging a safe applicant means a human takes a second look.

So I lowered the threshold to 0.15. Anyone the model gives a 15% or higher chance of default is marked higher risk. That's what produces the 42% recall above, at the cost of precision: roughly one in four flagged applicants turns out to be a true defaulter.

That trade is deliberate. The model isn't making the final call; it's deciding who deserves a closer look, and for that job catching more of the risky cases matters more than being right every time it raises a hand.

What the model leans on

The top features by importance were the three external credit scores, followed by the price of the goods the loan is for and the credit amount itself. That's reassuring: the model relies mostly on signals a credit analyst would also trust, rather than on something accidental in the data.

Keeping training and the app identical

The preprocessing and the model are saved together as one pipeline with joblib. The Streamlit app loads that single object, so every prediction goes through exactly the same steps used in training. It's a small decision that rules out a whole class of "works in the notebook, wrong in the app" bugs.

What I'd do next

Precision at 25% is the obvious weakness. The next steps are calibrating the probabilities so that "15%" really means 15%, and choosing the threshold from an explicit cost of each error rather than by judgement.

The code, notebook and app are on GitHub, and there's a short write-up on the project page.

First published on sangamgupta.in.