An LLM routes our support tickets into three queues: general, billing and refund. I wanted an F1 score on the eval dashboard, so I called f1_score on 100 labeled tickets.

It crashed with this error: Target is multiclass but average='binary'. I added average='micro', the error went away, and the dashboard showed 0.85.

Accuracy on the same 100 tickets was also 0.85. I read that as two metrics agreeing. On single-label data, micro F1 and accuracy are the same calculation.

The default only works for two classes

f1_score defaults to average='binary'. That mode scores one positive class, picked with pos_label, and it only accepts two-class targets.

Three queues make a multiclass problem. So scikit-learn raises the error and asks you to pick micro, macro, weighted or None.

The error makes you choose how to combine the per-class scores. Each option measures something different, and I picked one without reading what it computed.

Micro F1 on single-label data is accuracy

Micro averaging pools the counts. It adds up true positives, false positives and false negatives across all classes, then computes one F1 from the totals.

In single-label classification, every wrong prediction counts twice. It is a false positive for the queue the model picked and a false negative for the queue the ticket belonged in. So the pooled false positives always equal the pooled false negatives.

That makes micro precision equal micro recall, and both equal accuracy. On my 100 tickets, 85 were routed right and 15 were wrong. Micro precision, micro recall, micro F1 and accuracy all came out at 0.85.

scikit-learn's classification_report leaves out the micro average row for single-label data (unless you pass only some of the labels), because that row would only repeat accuracy.

Macro F1 showed the queue I was failing

The 100 tickets were 80 general, 12 billing and 8 refund. When I printed one score per class with average=None, the per-class scores looked like this:

  • General: 78 of 80 routed right, F1 0.92.
  • Billing: 6 of 12 routed right, F1 0.60.
  • Refund: 1 of 8 routed right, F1 0.20.

The misses: both wrong general tickets went to billing, 5 billing tickets went to general and 1 to refund, and 7 refund tickets went to general.

Macro F1 gives each class the same weight and averages the three scores. It came out at 0.57.

Macro F1 showed what micro hid. Seven of the eight refund tickets were sitting in the general queue. Micro F1 could not show it, because general tickets made up 80 percent of the data and drowned out the rest.

Weighted F1 came out at 0.82. It weights each class by how many tickets it has, so the general queue dominates that one too.

Small slices break macro F1 in a different way

After I switched to macro, I started running it on daily slices. One quiet day had 10 tickets: 8 general, 2 billing and no refunds. One billing ticket went to general.

I passed labels=['general', 'billing', 'refund'] so every daily report showed the same three classes. Macro F1 for that day came out at 0.54.

Refund had no true tickets and no predictions that day. Its F1 is undefined. With the default zero_division='warn', scikit-learn sets that F1 to 0, prints a warning, and averages the 0 in. No refund tickets came in that day, so the router had nothing to miss.

Setting zero_division=np.nan drops undefined classes from the average. It has been available since scikit-learn 1.3. The same slice then scores 0.80.

One detail I had wrong: I expected a warning whenever a class gets no predictions. f1_score only warns when a class has no true labels and no predictions at all. If refund tickets exist and the router never predicts refund, refund's F1 is 0, with no warning. precision_score does warn in that case, so I check both.

What I would tell myself before that error

  • Picking average to make an error go away. Read what each option computes before you choose one.
  • Reporting micro F1 next to accuracy. On single-label data they are the same number.
  • Trusting one averaged score. Print average=None and look at each class first.
  • Passing labels= on small slices with the default zero_division. A class with no tickets counts as a 0 in the macro average. Use np.nan, or show the slice size next to the score.

If you want the full version, this piece covers the F1 formula, F-beta, threshold tuning and F1 for LLM classifiers, with scikit-learn code you can run.

Which average setting did you pick the first time, and why? For me it was micro, and the reason was an error message I wanted gone.