Not Even Root Can Move a Managed Interrupt

Back in April I wrote a SoftIRQ saturation and IRQ affinity runbook on this blog. Its third section covers what to do when interrupts pile onto one core: choose cores according to the card's queue count, map the IRQs onto those cores, verify the distribution with /proc/interrupts. There's also a sentence saying deliberate pinning is the safer option on critical production nodes.

Seven months later I ran that step end to end for the first time on my own machine, the server that hosts this blog. On twenty of the forty interrupt lines the mapping step isn't possible at all: the mask I write as root is refused, and the error that comes back says "I/O error" — as though something were wrong with a cable. On the other twenty, pinning works, and it works instantly. What doesn't work is the move people reach for first: widening the mask.

This article is the measurement of those two halves. In an October piece about rq_affinity I dropped a side observation that "all the interrupts land on one core" — that experiment ran on a different VM with 14 cores and a single hardware queue, and the subject there was which ledger completions get written into. I never looked at why the interrupt itself wouldn't move. I'm looking now.

The machine and the method

The measurements were taken on 10 November 2026 on the server hosting this blog: Ubuntu 24.04.4 LTS, kernel 6.8.0-147-generic, 18 vCPUs, QEMU/KVM (i440FX), storage on virtio-scsi (PCI 00:05.0), network on virtio-net (PCI 00:12.0), up for three weeks and one day. The irqbalance service is inactive — so there's no background process quietly balancing interrupts.

Since I didn't want to play with routing on a live server, the work started with a side-effect-free probe: write each line's own current mask back into itself. The value is identical, so nothing changes; if the write is refused, the errno comes back.

for d in /proc/irq/[0-9]*; do cur=$(cat "$d/smp_affinity") || continue if err=$( (printf '%s' "$cur" > "$d/smp_affinity") 2>&1 ); then printf '%-4s WRITE_OK\n' "${d##*/}" else printf '%-4s WRITE_FAIL %s\n' "${d##*/}" "$err" fi done 

Distributions were counted by hand too; subtracting the columns of a /proc/interrupts line between two snapshots is enough:

a=$(grep -E '^ *46:' /proc/interrupts); sleep 20; b=$(grep -E '^ *46:' /proc/interrupts) awk -v A="$a" -v B="$b" 'BEGIN{ split(A,x," "); split(B,y," "); for(i=2;i<=19;i++){ d=y[i]-x[i]; if(d>0) printf "CPU%d=%d ", i-2, d } print "" }' 

Two side effects, stated up front. First: on four old ISA lines with no driver attached (3, 5, 7, 13), effective_affinity_list read 0 before the probe and 0-17 after it. The reason sits in kernel/irq/manage.c, two lines above the if block quoted later on: irq_set_affinity_deactivated() copies the mask and updates the effective mask with it, without calling the chip driver at all, for an interrupt that isn't activated yet. Second: during the migration experiment further down, the NIC's receive interrupt spent a while on CPU3; when the experiment finished I put it back where it was, on CPU9, and verified the final state.

Twenty of forty refused the write

The result came out surprisingly sharp:

Eighteen of the twenty refusals belong to a single device: virtio1-request, the request queues of the virtio-scsi controller. Each one's mask is a single CPU — IRQ 27 only CPU0, IRQ 28 only CPU1, and so on to IRQ 44 on CPU17.

My first instinct was that the refusal might be because the value was identical. It isn't: I wrote 3 (CPU0 and CPU1) to IRQ 27 instead of its current mask, got the same error, and smp_affinity_list still read 0. The refusal is about the line's identity, not the value.

The remaining two lines are 0 and 2, the timer and the cascade. The gate in the code actually checks three things at once — that the line can be balanced, that its affinity isn't kernel-managed, and that noirqaffinity wasn't passed at boot — and I could not determine by measurement which of them these two tripped. Why I couldn't comes later.

Why eighteen lines refuse

The answer is in the block layer. Under sda's mq directory there are eighteen hardware queues, each with a cpu_list of exactly one core:

/sys/block/sda/mq/0/cpu_list → 0 (nr_tags=256) /sys/block/sda/mq/1/cpu_list → 1 (nr_tags=256) ... /sys/block/sda/mq/17/cpu_list → 17 (nr_tags=256) 

Every core has its own queue and every queue its own interrupt vector. The driver asked for that arrangement: when drivers/scsi/virtio_scsi.c looks up its queues it hands the kernel a placement descriptor (struct irq_affinity desc = { .pre_vectors = 2 }), and in response the virtio-pci layer adds the PCI_IRQ_AFFINITY flag and calls pci_alloc_irq_vectors_affinity(). The kernel's PCI documentation describes what that flag does in one sentence: it spreads the interrupts around the available CPUs.

Interrupts distributed this way are what the kernel calls managed, and it deliberately blocks user space from touching them. The write handler in kernel/irq/proc.c closes the door on its first line:

 if (!irq_can_set_affinity_usr(irq) || no_irq_affinity) return -EIO; 

And irq_can_set_affinity_usr() in kernel/irq/manage.c combines two conditions:

bool irq_can_set_affinity_usr(unsigned int irq) { struct irq_desc *desc = irq_to_desc(irq); return __irq_can_set_affinity(desc) && !irqd_affinity_is_managed(&desc->irq_data); } 

This also explains the most confusing part of the table: on the same machine the virtio1 lines refuse while the virtio2 lines accept, even though both are virtio. The difference is in the driver. The network driver, drivers/net/virtio_net.c, hands over no such descriptor when it looks up its queues — the irq_affinity struct doesn't appear anywhere in the file — and writes the placement itself through virtnet_set_affinity(). With no descriptor there's no PCI_IRQ_AFFINITY, the interrupt isn't considered managed, and the procfs door stays open. So there's no rule saying "modern device, therefore managed"; what decides is whether the driver handed over that descriptor.

That has a further practical consequence: a mask you write by hand onto the network interrupts can be rewritten by the driver's own virtnet_set_affinity() call when the queue count changes or a core goes offline. Your pinning isn't a lasting contract.

Why managed interrupts exist, and how that reaches you

I could have waved this away as "don't break the design," but the kernel's own rationale is more concrete. managed_irq.rst, added to the documentation in April 2026, explains it: an NVMe device with 128 queues typically requests one interrupt per queue, two such devices make 256 interrupts, and on x86 there are only 256 vectors per CPU with some already reserved. Normally this isn't a problem, because the interrupts are spread across cores — but when the system suspends, every secondary CPU is taken offline and all interrupts migrate to the one core still running. That's where the vector space runs out and the suspend fails. Managed interrupts exist precisely to prevent that migration.

The operational face of this is in the same document: when a core in the mask goes offline the interrupt moves to the next core in the mask, and when the last core in the mask goes down the interrupt is shut down, with the driver required to have quiesced the matching queue. On my machine each queue's mask is a single core, so that "last core" case is the same thing as taking that core offline.

The same page also gives the supported way to keep managed interrupts away from particular cores: isolcpus=managed_irq,<mask> at boot. It's best-effort — if the automatically computed mask has no online core outside the avoided set, the setting has no effect. So when a managed interrupt lands on a hot spot you aren't empty-handed; the lever just isn't in /proc, it's on the boot line.

"I/O error" was the wrong word, and the kernel agreed

EIO — "Input/output error" — makes an administrator think of hardware trouble. What's actually happening is a permission matter. Others noticed before me.

First, irqbalance. It's the software that actually writes these files, and it translates the error internally; in activate.c the first line of the write-error branch reads, verbatim:

error: /* Use EPERM as the explaination for EIO */ errsave = (errsave == EIO) ? EPERM : errsave; 

There's more: if the error is in the permanent class, irqbalance drops that interrupt to BALANCE_NONE, logs "IRQ %i affinity is now unmanaged", and never bothers with it again. On my server it's off anyway; had it been running, it would have quietly given up on those twenty lines.

Second, the kernel itself. Two patches landed a day apart in August 2024, and both shipped in v6.12:

• eb29369fa543 changed the returned error from EIO to EPERM; the commit message states the rationale plainly.
• c7718e5c76d4 is the more radical one: the kernel already knows at allocation time whether an interrupt can be set from user space, so it should create the procfs files for the ones that can't as read-only (0444).

The consequence of the second patch: on a newer kernel a single ls -l shows which interrupts you can't move. On my 6.8 it doesn't, because all forty files are 0644. The twenty files that refuse every write look exactly as writable as the rest.

Rather than assume the version map, I scanned it. The value returned in kernel/irq/proc.c is EIO on v6.9, v6.10 and v6.11, and EPERM from v6.12 through v7.3 and on master. Then I saw the fixed behaviour in a lab: an Ubuntu 25.10 VM on my Mac running kernel 6.17.0-40-generic, where 16 of its 36 interrupt lines are mode 0444, the block devices' request queues among them. Writing a line's own mask to one of those as root gets you Operation not permitted.

A small aside: that fix also made irqbalance's translation line redundant. The kernel now returns EPERM directly, and since the file is read-only the write usually isn't even attempted.

The diagnosis — the answer to "why won't this line move?" — is two years and one LTS generation away from my server's kernel. On the same machine apt-cache policy linux-generic-hwe-24.04 currently offers 7.0.0-42 as the candidate. One package, but jumping a kernel generation on a production server isn't a decision you make for the convenience of ls -l, and I didn't make it for this article.

The twenty that accept: a narrow mask moves them, a wide mask does nothing

Now the other half. On all twenty lines that accept the write, smp_affinity_list reads 0-17 — on paper, "this interrupt may land on any of eighteen cores." But effective_affinity_list on those lines shows a single core, and the counters back the effective mask. Three 20-second runs on the two busiest lines, the network card's receive and transmit queues:

44,993 interrupts in total, with seventeen columns reading exactly zero for each line. In the same minute the local timer interrupt LOC was spread across all eighteen cores, so the machine is running multi-core.

So what happens if I narrow the mask? I didn't want to write this up without measuring it, because that's the real question here. I wrote 8 (CPU3 only) to IRQ 46, then 200 (CPU9 only), counting for ten seconds at each step:

Three rows, three separate statements. B: a narrow mask works, and it works instantly — by the first read after writing the file, the effective core had already changed. On a line taking a few hundred interrupts a second there's nothing to wait for. C: reopening the mask to eighteen cores does nothing at all; the interrupt stays on CPU9, neither spreading nor being re-picked. A: it was already sitting on CPU3 with a wide mask, because I had put it there in the previous step and nobody moved it back — irqbalance is off, and the kernel has no business redistributing on its own.

That lets me rename the mask properly: smp_affinity is a candidate set, not a distribution order. An interrupt delivered over MSI-X goes to a single target; the kernel picks one core out of the set you gave it. Narrow the set and you force the choice — that works. Widen it and you've asked for nothing. April's runbook saying "deliberate pinning" turns out to have been right for exactly this reason; what's wrong is the instinct that opening the mask splits the load.

The exception the floppy drive taught me

There is a case where the migration really doesn't happen, and hardware this machine doesn't have showed it to me. IRQ 6 belongs to floppy: there's no floppy drive, the line is there, and since boot — three weeks and a day — it has counted three interrupts, all on CPU16.

start: smp_affinity_list=0-17 effective_affinity_list=16 echo f → OK smp_affinity_list=0-3 effective_affinity_list=16 echo 2 → OK smp_affinity_list=1 effective_affinity_list=16 restore → OK smp_affinity_list=0-17 effective_affinity_list=16 

All three writes succeed, and the mask file reads back exactly what I wrote each time. The effective core doesn't move: CPU16, CPU16, CPU16. What happened instantly on IRQ 46 never happens here, because the difference isn't in the write, it's in the interrupt.

The mechanism is written down in kernel/irq/manage.c. The write request lands in irq_set_affinity_locked(), and if the chip cannot move the interrupt in process context, the change isn't applied immediately, it's put pending:

 if (irq_can_move_pcntxt(data) && !irqd_is_setaffinity_pending(data)) { ret = irq_try_set_affinity(data, mask, force); } else { irqd_set_move_pending(data); irq_copy_pending(desc, mask); } 

The place where a pending move is applied is the path the interrupt itself travels: apic_ack_edge() in arch/x86/kernel/apic/vector.c calls irq_complete_move() and then goes on to apic_ack_irq(); the irq_move_irq() call lives there, and the pending mask finally takes effect in irq_move_masked_irq() in kernel/irq/migration.c. In other words, an interrupt migrates to its new core only the next time it fires. On a network queue taking hundreds of interrupts a second that takes microseconds; on a line that fired three times in three weeks, when "next time" happens is anybody's guess.

So why did the mask file show what I wrote? That's in proc.c: if a move is pending, procfs prints the pending mask instead of the real one.

 mask = desc->irq_common_data.affinity; #ifdef CONFIG_GENERIC_PENDING_IRQ if (irqd_is_setaffinity_pending(&desc->irq_data)) mask = desc->pending_mask; #endif 

That display is still in today's master, only the helper names have changed. The decision gate itself, though, has not stayed put: v6.14 introduced the IRQCHIP_MOVE_DEFERRED flag (a648eb3a3f79) and x86 was converted to that model (7d04319a05ab). Deferral is now a property of the chip rather than a per-interrupt marking, and on master the IO-APIC and MSI chips carry the flag. The "applied immediately if remapping is present" distinction that holds on my 6.8 doesn't hold in the same shape on newer kernels.

I can't ask the kernel this question

Earlier I said I couldn't determine which condition made those two lines refuse. Here's why. The kernel has a debug interface that exposes per-interrupt flags, but it depends on a build option, and my server's configuration has it off:

# CONFIG_GENERIC_IRQ_DEBUGFS is not set 

That's why there's no /sys/kernel/debug/irq directory — debugfs is mounted, there are forty-odd other directories inside it, this one isn't there. So on 6.8 the only answer I can get to "is this interrupt managed?" is the errno of a write attempt. The question isn't free: to get the answer you have to touch the thing. That's why the probe had to be side-effect-free. And the kernel's own documentation recommends exactly this debugfs file for confirming a managed interrupt's state — a recipe that doesn't run on my box as long as the distribution doesn't compile it in.

The documentation: one page from 2011, one from this April

The page that comes to mind first for this knob is Documentation/core-api/irq/irq-affinity.rst. Seventy lines, and reading it surprised me twice.

First, its example: on an eight-CPU box the IO-APIC-level eth1 line spreads interrupts across CPU0-3 once the mask is set to 0f (1068, 1785, 1785, 1783), and moves to the other four with f0. On my machine, opening the mask spreads nothing. Both are correct — one describes an IO-APIC level-triggered card, the other a virtio device using MSI-X — but the document doesn't help you tell which situation you're in. Second, the omissions: neither the effective_affinity file nor managed interrupts appear on that page. The only sentence saying writes aren't entirely free is the warning that you're not allowed to switch off all CPUs.

The page's stillness got measured too: the difference between v6.8 and today's master is four lines — "cpu" capitalised to "CPU", and a missing grep restored in the example. The last addition to its content was the smp_affinity_list paragraph in 2011; all five touches since then were spelling, formatting and a move.

But ending the story there would be unfair, because the kernel closed this gap seven months ago. On 1 April 2026, commit bb6a85b4b652 added a new page in the same section: managed_irq.rst, listed directly beneath irq-affinity. Both of this article's findings are in it: that the affinity of managed interrupts is handled entirely in the kernel and cannot be modified through the /proc interfaces, and example output where smp_affinity_list and effective_affinity_list diverge. The example hardware is too familiar to be a coincidence: a QEMU instance booted with -device virtio-scsi-pci, three management interrupts and one interrupt per queue.

The only problem is that the page doesn't exist where I work. It's absent in v7.0 and present from v7.1 on. The official account of managed interrupts is three generations ahead of the kernel on this server. The knowledge exists; it just isn't on the page I was reading or in the kernel I'm running.

What the real lever is on this machine

The measurement showed that the "open the mask, let the load spread" move is empty on this box. The right doors got measured as well.

The network side. ethtool -l eth0 says one thing: Combined: 1, with a pre-set maximum of 1. The device has a single queue pair; there's no second interrupt vector to spread. Fiddling with interrupt affinity on a card like that is like looking for a second cashier in a shop with one till. If the load genuinely saturates one core, the lever is on the software side: RPS. That's off too — /sys/class/net/eth0/queues/rx-0/rps_cpus reads 00000. I haven't touched it, because I have not measured a bottleneck right now.

The block side. Nothing to do there, because it's already done. The sixty-second delta for the eighteen virtio1-request lines:

19,380 interrupts in total, not one of the eighteen lines saw a second core, and the core each landed on matches its cpu_list exactly. It's the mirror image of the network table's "mask of eighteen, reality of one": here the mask is one and reality is one, and because the two agree, nobody gets confused.

The list I took away for anyone who has to look at this knob in production:

1. Read effective_affinity first. smp_affinity is intent, effective_affinity is state. Without seeing the gap it's very easy to conclude "the mask is wide, so it must be spreading."
2. Verify with the counters. Learn how many cores a line lands on from the /proc/interrupts delta, not from the mask.
3. On kernel 6.12 or newer, run ls -l /proc/irq/*/smp_affinity. The 0444 ones can't be moved, and asking costs nothing.
4. On anything older, probe with each line's own mask. The value is identical so nothing changes, and the errno still arrives.
5. If you want spreading, widening the mask won't give it to you. The measurement is unambiguous: a wide mask neither distributes nor re-picks. Spreading comes from queue count (ethtool -L, the device's num_queues) or from RPS/XPS.
6. Don't fight the lines that refuse; the right lever is the boot line. isolcpus=managed_irq,<mask> tries to keep managed interrupts off specific cores, and irqaffinity= sets the default for the management interrupts.
7. Know the cost of offlining a core under managed interrupts. When the last core in the mask goes offline, the interrupt is shut down.
8. A change on a quiet line may never be applied. The pending move waits for the next interrupt; if it doesn't come, nothing happens.

Conclusion

What I take from this is bigger than one knob. Under /proc and /sys there are two kinds of file: the ones holding what I asked for, and the ones holding what the system is doing. Most of the time they show the same number, so we treat them as one file. The gap between smp_affinity and effective_affinity is exactly this, and it only surfaces when things aren't ordinary — while a move is pending, when the interrupt never fires, when the device distributed its own queues. Which is precisely when you reached for the setting.

As for April's runbook: the "deliberate pinning" advice came out of the measurement intact — a narrow mask genuinely works, and works instantly. What was missing is where that advice cannot be applied — on the interrupts of a driver that asks for a vector per queue, the step never starts, and it tells you so with an "I/O error". The me of seven months ago didn't know that, because I hadn't run that article all the way through on my own server.

And third: the kernel wasn't wrong here, it had just picked the wrong word; it fixed the word two years ago and documented the gap this April. Both are ahead of the version I'm running. Getting the name of an error right does as much work as adding a feature — provided you can reach the version where that name appears.

Official Sources

• SMP IRQ affinity — Linux kernel documentation
• Affinity managed interrupts — managed_irq.rst (added April 2026)
• MSI howto: pci_alloc_irq_vectors_affinity and PCI_IRQ_AFFINITY
• kernel/irq/proc.c — write_irq_affinity and show_irq_affinity (v6.8)
• kernel/irq/manage.c — irq_can_set_affinity_usr and the pending move (v6.8)
• drivers/scsi/virtio_scsi.c — struct irq_affinity desc (v6.8)
• eb29369fa543 — genirq/proc: change the return value from EIO to EPERM
• c7718e5c76d4 — genirq/proc: read-only files for interrupts that cannot be set
• bb6a85b4b652 — Documentation: Add managed interrupts
• irqbalance activate.c — the branch that logs EIO as EPERM