Hey all 👋
Last episode was an RCA with seven root causes and a port number. This one is an RCA with one root cause, five wrong theories, and a cluster that reported itself healthy for thirty-eight hours while doing absolutely nothing.
State of the lab going in: two nodes, Flux in charge, Airflow on a public domain through a Cloudflare tunnel, nightly backups, DHCP reservations finally ticked. Episode 10 ended with "the router remains the weakest member of the team." I would like to amend that statement. The router has competition.
Here's what happened, in the order I found out, which is not the order it happened.
🚨 Symptom: The Tunnel Is Down
Sunday night, 10 PM. Cloudflare's Zero Trust dashboard, which has shown two happy connectors since Episode 9, says the tunnel is down. No connectors. Nothing.
First look at the cluster:
$ kubectl get nodes NAME STATUS ROLES AGE VERSION dell NotReady <none> 64d v1.35.6 msi Ready <none> 64d v1.35.6 Dell is down. Has been for two and a half days, it turns out. And the pods tell the failover story exactly the way the textbook says it should go:
$ kubectl -n cloudflared get pods -o wide NAME READY STATUS RESTARTS AGE NODE cloudflared-6cf54796d6-42rg9 1/1 Running 0 2d16h msi cloudflared-6cf54796d6-7g425 1/1 Terminating 0 5d9h dell cloudflared-6cf54796d6-jkt44 1/1 Running 0 2d16h msi cloudflared-6cf54796d6-t567p 1/1 Terminating 0 5d9h dell Dell died, the old pods went Terminating, two fresh replicas came up on msi 2d16h ago. Running. 1/1. Zero restarts. Kubernetes did its job.
Small aside before we continue. Episode 9 proudly said: two replicas with pod anti-affinity so the tunnel survives a node dying. Look at the ages. Both old replicas were on dell. Anti-affinity on a two-node cluster is a preference, and the scheduler preferred dell twice. The tunnel survived the node dying anyway, because failover worked. It just didn't survive what came next.
So the question was simple. Two healthy connectors on msi, dashboard says zero. Why?
🔬 Hypothesis 1: The Failover Path Was Never Tested
Both replicas had always lived on dell. This was the first time cloudflared had ever run from msi. Maybe msi's egress was different: UDP 7844 blocked, QUIC failing, DNS for argotunnel.com not resolving from msi's pods.
The test: read the pod's log. The fix if true: TUNNEL_TRANSPORT_PROTOCOL=http2 to force TCP, or fix whatever was eating UDP.
What happened instead:
$ kubectl -n cloudflared logs cloudflared-6cf54796d6-42rg9 Error from server: Get "https://192.168.0.201:10250/containerLogs/...": tls: failed to verify certificate: x509: certificate is valid for 192.168.68.210, not 192.168.0.201 Episode 10, Sidequest 2, verbatim, except on the other node. When I moved subnets, I fixed dell's kubelet certificate because that's where the pod I wanted to read was. msi's kubelet had the same stale certificate the entire time, and nobody noticed because nobody asked msi for logs. kubectl get doesn't touch the kubelet. kubectl logs does.
Fine. The host has the logs on disk. Over ssh:
sudo tail -n 80 /var/log/pods/cloudflared_cloudflared-6cf54796d6-42rg9_*/cloudflared/0.log And the log was a complete short story:
2026-10-31T01:11:58Z INF Registered tunnel connection connIndex=1 ... location=blr03 protocol=quic 2026-10-31T01:11:58Z INF Registered tunnel connection connIndex=2 ... 2026-10-31T01:11:59Z INF Registered tunnel connection connIndex=3 ... 2026-10-31T01:12:03Z INF | SUMMARY: Environment is healthy. cloudflared will use 'quic' as primary protocol. | ... 2026-11-01T07:24:54Z WRN Serve tunnel error error="... timeout: no recent network activity" connIndex=0 2026-11-01T07:24:54Z WRN Serve tunnel error error="... timeout: no recent network activity" connIndex=3 2026-11-01T07:24:54Z WRN Serve tunnel error error="... timeout: no recent network activity" connIndex=2 2026-11-01T07:24:54Z ERR Serve tunnel error error="... timeout: no recent network activity" connIndex=1 ... 2026-11-01T07:25:19Z INF Initiating graceful shutdown due to signal terminated ... 2026-11-01T07:25:22Z ERR no more connections active and exiting 2026-11-01T07:25:22Z INF Tunnel server stopped Hypothesis 1, dead. Failover worked perfectly. The msi replica registered four connections on Friday morning and every pre-check passed. Then on Saturday at 12:54 PM (the logs are UTC; I'm not), all four connections timed out in the same second, a SIGTERM arrived twenty-five seconds later, and the process exited cleanly.
And then nothing. No new container. The log file ends there. For thirty-eight hours, the API said Running, 0 restarts about a process that had logged its own obituary.
That changed the question. It was no longer "why is the tunnel down." It was "why did the kubelet not restart a dead container, and why is it lying about it?"
📡 Interlude: Who Killed It
Four connections dying in the same second is not Cloudflare. That's the floor falling out from under the network. msi's journal for that minute:
Nov 01 07:24:49 msi wpa_supplicant[760438]: wlo1: CTRL-EVENT-BEACON-LOSS Nov 01 07:24:50 msi wpa_supplicant[760438]: wlo1: CTRL-EVENT-DISCONNECTED bssid=... reason=4 locally_generated=1 Nov 01 07:25:32 msi microk8s.daemon-kubelite[1122435]: ... <- kubelite starting Nov 01 07:25:43 msi microk8s.daemon-kubelite[1122802]: ... <- and again Nov 01 07:25:53 msi microk8s.daemon-kubelite[1123187]: ... <- and again Nov 01 07:25:54 msi wpa_supplicant[760438]: wlo1: CTRL-EVENT-CONNECTED Nov 01 07:26:55 msi microk8s.daemon-kubelite[1123923]: ... <- fourth time's the charm Nov 01 07:28:19 msi kernel: wlo1: deauthenticated from ... (Reason: 7=CLASS3_FRAME_FROM_NONASSOC_STA) Nov 01 07:28:19 msi kernel: iwlwifi 0000:00:14.3: 0x004D9C74 | branchlink2 Wi-Fi power save. The laptop's radio napped through too many beacons, decided the router was gone, and disconnected itself (reason=4 locally_generated=1, which is the card saying "this was my idea"). When the address vanished, so did the thing dqlite is bound to. Episode 5 explained this exact mechanism: a single-node MicroK8s keeps its datastore on loopback and is immune to this, and the day the Dell joined, dqlite moved to the LAN address and became a hostage of the Wi-Fi card. kubelite crash-looped four times in two minutes. Then the router kicked msi off twice more (reason=7, the router's idea this time), with an Intel firmware crash dump for garnish.
Somewhere in that mess, cloudflared got its SIGTERM. I never pinned down who sent it; my best guess is a liveness probe doing its job during the chaos. It doesn't matter much. What matters is that the kubelet came back after the fourth restart and then did nothing, for a day and a half.
The Episode 5 fix for the symptom, applied on the spot:
sudo iw dev wlo1 set power_save off The actual fix is a cable. We'll get there.
🪪 Blocker: The Certificate, Again
Before any more theories I needed kubectl logs and exec working on msi, because a debugging session without them is archaeology. Episode 10's trick was --rotate-server-certificates=true: let the kubelet request a fresh cert, approve the CSR, done. On msi it produced a different error:
Error from server: Get "https://192.168.0.201:10250/containerLogs/...": remote error: tls: internal error $ kubectl get csr No resources found tls: internal error is the kubelet saying "rotation is on, I have nothing to serve yet, and I'm not going to serve the old file either." And it never filed a CSR. No error about why. Just a kubelet politely waiting for a certificate it never asked for. Hold that thought, because in hindsight it's the same bug wearing a different hat.
I stopped debugging rotation and signed the certificate by hand with the cluster CA. Same private key, new SAN list:
cd /var/snap/microk8s/current/certs sudo cp -p kubelet.crt kubelet.crt.bak printf '%s\n' \ 'basicConstraints=CA:FALSE' \ 'keyUsage=critical,digitalSignature,keyEncipherment' \ 'extendedKeyUsage=serverAuth' \ 'subjectAltName=DNS:msi,IP:192.168.0.201,IP:127.0.0.1' > /tmp/kubelet-ext.cnf sudo openssl req -new -key kubelet.key -subj "/CN=msi" -out /tmp/kubelet.csr sudo openssl x509 -req -in /tmp/kubelet.csr -CA ca.crt -CAkey ca.key \ -CAcreateserial -CAserial /tmp/kubelet-ca.srl -days 3650 -sha256 \ -extfile /tmp/kubelet-ext.cnf -out /tmp/kubelet.crt sudo install -m 660 -o root -g microk8s /tmp/kubelet.crt kubelet.crt sudo sed -i '/^--rotate-server-certificates=true$/d' /var/snap/microk8s/current/args/kubelet sudo snap restart microk8s.daemon-kubelite $ echo | openssl s_client -connect 192.168.0.201:10250 2>/dev/null | openssl x509 -noout -ext subjectAltName X509v3 Subject Alternative Name: DNS:msi, IP Address:192.168.0.201, IP Address:127.0.0.1 Certificate fixed. kubectl logs again:
$ kubectl -n kube-system logs deploy/coredns --tail=3 Error from server (NotFound): the server could not find the requested resource ( pods/log coredns-78894c95f4-l8n8j) A 404. From the kubelet. The TLS handshake succeeded, the API server asked msi's kubelet for coredns's logs, and the kubelet said "never heard of it." About a pod the API says has been running on msi for 64 days.
That's not a certificate problem. That's the kubelet not knowing what it's running.
🔬 Hypothesis 2: The Kubelet Has Lost Its Pods
The test: ask the kubelet directly for its pod list, through the API server's node proxy, and compare with what the API thinks.
sudo microk8s kubectl get pods -A --field-selector spec.nodeName=msi -o name | wc -l # 34 sudo microk8s kubectl get --raw /api/v1/nodes/msi/proxy/pods # {"kind":"PodList", ..., "items": null} Thirty-four pods in the API. Zero in the kubelet. Not "some missing." Null. And the kubelet's journal, every ten seconds, since a very specific moment:
Nov 01 07:26:57 msi kubelite: E1101 07:26:57 kubelet_node_status.go:392] "Error getting the current node from lister" err="node \"msi\" not found" 07:26:57. Two seconds after the fourth kubelite restart on Saturday. The kubelet can't find its own node, and it's been saying so 15,000 times while kubectl get nodes says msi Ready.
Confirmed, then: the kubelet's view of the cluster is empty. Everything downstream follows. It won't restart a container belonging to a pod it doesn't know. It won't write status for it, so the API keeps showing the last status it ever wrote (the frozen Running, 0 restarts). It answers log requests with 404. And meanwhile containerd, which doesn't care about any of this, keeps running whatever was already running:
$ sudo microk8s ctr tasks ls | awk 'NR>1{print $3}' | sort | uniq -c 59 RUNNING Airflow was fine. Postgres was fine. Fifty-nine containers happily running with no supervisor. The only things missing were the ones that had died, which nobody was around to restart.
But "the kubelet's view is empty" is a symptom. The question is why.
🔬 Hypothesis 3: The API Server's Watch Cache Is Empty
Quick mechanism, because the rest of the post depends on it. The API server has two ways to answer a read. A plain GET goes to the datastore. But every component that watches the cluster (the kubelet, the controller-manager, the scheduler, Flux) reads through the watch cache: an in-memory copy of the datastore that the API server keeps current by watching the datastore itself. Thousands of watchers, one datastore watch. That's what lets Kubernetes scale. You can also read from it on purpose, by asking for resourceVersion=0.
If that cache were empty, you'd get exactly this: kubectl sees a healthy cluster (datastore), every component sees nothing (cache).
The test: the same question, both ways. The fix if true: restart kubelite, since the cache is rebuilt on every API server start.
$ sudo microk8s kubectl get --raw '/api/v1/pods?fieldSelector=spec.nodeName%3Dmsi' | ... 34 $ sudo microk8s kubectl get --raw '/api/v1/pods?fieldSelector=spec.nodeName%3Dmsi&resourceVersion=0' | ... 34 Dead. The cache had all 34 pods. (I had also restarted kubelite twice by then, so an empty cache would have refilled anyway. Keep that in mind; it comes back.)
🔬 Hypothesis 4: The Kubelet's Credentials Are Broken
The API server answers me fine. Maybe it doesn't answer the kubelet. Episode 7 was an entire episode about the control plane being forbidden from reading its own objects, so this had form.
The test: replay the kubelet's exact startup queries using the kubelet's own kubeconfig. The fix if true: whatever the error says, probably certificates or RBAC.
$ K="sudo /snap/microk8s/current/kubectl --kubeconfig /var/snap/microk8s/current/credentials/kubelet.config" $ $K auth whoami Username system:node:msi Groups [system:nodes system:authenticated] $ $K get --raw '/api/v1/nodes?fieldSelector=metadata.name%3Dmsi&resourceVersion=0' | ... 1 ['msi'] $ $K get --raw '/api/v1/pods?fieldSelector=spec.nodeName%3Dmsi&resourceVersion=0' | ... 34 $ $K get --raw '/api/v1/pods?fieldSelector=spec.nodeName%3Dmsi&watch=1&resourceVersion=0' | head -c 200 {"type":"ADDED","object":{"kind":"Pod", ... Dead. The kubelet's identity gets the node, gets all 34 pods, and can open a watch. Every request the kubelet could make works when I make it.
So the kubelet isn't making them.
🔬 Hypothesis 5: The Kubelet's Informers Are Failing
Each component keeps its own local cache of the objects it cares about (an informer): one LIST to fill it, then a WATCH to keep it current. The kubelet won't touch pods until its informers have loaded once. If the LIST or WATCH were failing, the kubelet would log it and retry. Loudly.
The test: grep for reflector errors. The fix if true: depends on the error, but at least there'd be one.
$ sudo journalctl -u snap.microk8s.daemon-kubelite --since "90 min ago" \ | grep -E 'reflector|failed to list|Failed to watch|forbidden|deadline exceeded' (nothing) Dead, in a way that was more informative than an error would have been. Nothing was failing. And the controller-manager, which runs in the same kubelite process, told me what "nothing" looked like from the inside:
$ sudo journalctl -u snap.microk8s.daemon-kubelite --since "90 min ago" | grep -E 'Waiting for caches to sync|Caches are synced' Nov 02 20:25:02 msi kubelite: I1102 20:25:02 shared_informer.go:370] "Waiting for caches to sync" Nov 02 20:25:02 msi kubelite: I1102 20:25:02 shared_informer.go:370] "Waiting for caches to sync" Nov 02 20:25:02 msi kubelite: I1102 20:25:02 shared_informer.go:370] "Waiting for caches to sync" ... forty of these, all from the same second ... Fifty minutes after startup. Forty controllers waiting for caches to sync. Zero Caches are synced. Every informer in the entire control plane was stuck in the waiting room, not erroring, just waiting. Which, incidentally, is why kubectl rollout restart deployment/cloudflared an hour earlier had done precisely nothing: the deployment controller was in the same waiting room.
This also explains the certificate rotation that never filed a CSR. The kubelet's cert manager needs the node informer to have synced before it can build the request. Same wait, different symptom.
Four hypotheses down. Every test says the API server is fine, the data is fine, the credentials are fine, nothing is failing, and nothing is happening.
🧊 Hypothesis 6: The Cache Isn't Empty. It's Frozen.
Here's the hole in Hypothesis 3. When an informer starts, it doesn't ask for "whatever the cache has." It asks for the current state, and the API server, before answering, waits for its watch cache to catch up to the datastore's latest revision. A fresh snapshot, guaranteed. My resourceVersion=0 test never asked for that guarantee. It asked for the stale copy and got it.
A frozen cache, one that took a snapshot and then stopped receiving updates, would pass my test and hang every informer forever. And my test couldn't tell frozen from live because nothing in the cluster was changing, because everything that changes things was stuck waiting on the cache. Circular. Beautiful. Infuriating.
To catch it I needed an object that was still being written. The kubelet renews its node lease every ten seconds, straight to the datastore, informers or no informers.
The test: read the lease from the datastore, then from the cache.
$ date -u Mon Nov 2 21:19:24 UTC 2026 $ ... leases/msi -> rv 3815025 renewTime 2026-11-02T21:19:19Z $ ... leases/msi?resourceVersion=0 -> rv 3810400 renewTime 2026-11-02T20:23:53Z Confirmed. The datastore's copy was five seconds old. The cache's copy was fifty-five minutes old, from 20:23:53, which is three seconds before I had restarted kubelite for the certificate. The cache took one snapshot at startup and then received nothing. Four thousand six hundred revisions behind and counting.
Every informer in kubelite was waiting for a cache to catch up to a datastore it had stopped listening to.
And why did restarting kubelite not fix it, twice? Because the cache is fed by a watch stream from k8s-dqlite, which is its own snap service:
$ systemctl show snap.microk8s.daemon-k8s-dqlite -p ActiveEnterTimestamp ActiveEnterTimestamp=Wed 2026-10-28 07:07:22 UTC Running since the Episode 10 fix. The one component in the whole chain that had never restarted. And its log from Saturday, during the Wi-Fi hole:
Nov 01 07:32:33 k8s-dqlite: [go-dqlite] attempt 10: server 192.168.0.201:19001: dial: connect: network is unreachable Nov 01 07:32:33 k8s-dqlite: error in txn: update transaction failed for key /registry/longhorn.io/nodes/longhorn-system/msi: context canceled Nov 01 07:32:42 k8s-dqlite: error while range on /registry/serviceaccounts/: query (try: 0): context canceled The datastore, unable to reach its own address, cancelling its own queries. Episode 10's dial ... i/o timeout to your own old IP, except this time the IP was correct and simply didn't exist for ninety seconds.
k8s-dqlite is really kine, a shim that makes dqlite speak etcd's API. kine finds new writes by polling its table and pushes them to anyone watching, including the API server's cache. Somewhere in that context canceled storm, the push stopped, while reads and writes carried on working. Every kubelite restart after that got a fresh cache with a perfect snapshot and a dead feed. I never found the exact line of kine that wedged. I found the fix, which was good enough at 3 AM.
🔧 The Fix
Restart the datastore, with the API server stopped so it isn't holding connections mid-stream, then bring the API server back to build a cache on a live feed. Containers keep running throughout; containerd doesn't care about any of this.
sudo microk8s dbctl backup /var/backups/microk8s/pre-restart-$(date +%F) # paranoia is a feature sudo snap stop microk8s.daemon-kubelite sudo snap restart microk8s.daemon-k8s-dqlite sleep 15 sudo snap start microk8s.daemon-kubelite Ninety seconds later:
$ ... leases/msi -> rv 3817920 renewTime 2026-11-02T21:50:45Z $ ... leases/msi?resourceVersion=0 -> rv 3817920 renewTime 2026-11-02T21:50:45Z $ journalctl ... | grep -c 'Caches are synced' 58 $ journalctl ... | grep 'Watching apiserver' Nov 02 21:49:13 msi kubelite: I1102 21:49:13 apiserver.go:52] "Watching apiserver" $ ... /api/v1/nodes/msi/proxy/pods -> 34 Same revision from both read paths. Fifty-eight controllers synced. The kubelet watching the API server again. Thirty-four pods known. And:
$ kubectl -n cloudflared get pods NAME READY STATUS RESTARTS AGE cloudflared-6cf54796d6-42rg9 1/1 Running 1 (38h ago) 2d20h cloudflared-6cf54796d6-jkt44 1/1 Running 1 (38h ago) 2d20h RESTARTS 1 (38h ago). The kubelet woke up, found two dead containers, restarted them, and for the first time in a day and a half wrote a status that was true. The "38h ago" is when the previous container died: Saturday, 12:55 PM, exactly where the log file ended.
$ kubectl -n cloudflared logs cloudflared-6cf54796d6-42rg9 --tail=5 INF Registered tunnel connection connIndex=0 ... INF Registered tunnel connection connIndex=1 ... INF Registered tunnel connection connIndex=2 ... INF Registered tunnel connection connIndex=3 ... Dashboard: Healthy. Two connectors. Airflow back on its domain.
🧠 What This Taught Me
- A cluster has two read paths, and they can disagree for days.
kubectl getreads the datastore. Every controller reads the watch cache. When the cache freezes, you get a cluster where every human check passes and every automated action silently stops. Nothing errors, because waiting isn't an error. - "Running, 0 restarts" is a claim the kubelet made once. Pod status is written by the kubelet. If the kubelet stops writing, the API keeps serving the last thing it wrote, indefinitely. Status is a snapshot, not a heartbeat.
- Every canary I had lied. Episode 7's
readyzwas green. Episode 7's coredns was1/1(it was frozen there). The controller-manager lease was even renewing (leader election doesn't need informers). The one thing that told the truth was a renewTime read two ways. New laminated card:kubectl get lease -n kube-node-lease <node>with and withoutresourceVersion=0. They must match. - Restarting the thing that's stuck doesn't help when the stuck thing is downstream. I restarted kubelite twice. The feed it depended on lived in a process I hadn't touched since October. Find the oldest PID in the chain.
- Hypotheses die by test, not by argument. Five theories, each plausible, each killed by one command. The frozen cache only became visible when I stopped testing content and started testing freshness, with an object that was still moving.
- Anti-affinity on two nodes is a suggestion. Both tunnel replicas lived on the same node for five days. The fix is a
topologySpreadConstraint, which is a rule, not a wish. - Joining the second node moved the datastore onto Wi-Fi, and Wi-Fi power save is a slow-motion power cut. Episode 5 said this about the router dying. Turns out the laptop can do it to itself, from the inside, whenever the radio feels sleepy.
power_save offis on now. The real fix is a cable into msi, and I've stopped pretending otherwise. - The certificate was never the problem, and I'd have been lost without fixing it. The stale kubelet cert blocked
logsandexec, and the rotation trick from Episode 10 failed because of the same frozen cache I hadn't found yet. Hand-signing it with the cluster CA took five minutes and unblocked everything after.
🚀 What's Next
Dell is still NotReady, with its ghost pods stuck in Terminating and a triggerer StatefulSet that won't schedule a replacement until the old pod is confirmed dead. That's its own evening and its own post. After that: a spread constraint and a /ready liveness probe on the tunnel pods, so a dead connector gets restarted instead of memorialized. And a monitoring question I don't have a good answer to yet: what do you alert on when the cluster's own health signals are the thing that froze?
💬 Final Thoughts
For thirty-eight hours my cluster was Schrödinger's: alive if you asked the datastore, dead if you asked anything that actually does work, and perfectly consistent with itself the whole time. The Wi-Fi card took a nap, the datastore cancelled its own queries, the cache stopped listening, and every controller in the building sat down to wait for an update that was never coming. Nobody errored. Nobody complained. Fifty-nine containers kept running on autopilot with no one at the controls, which is either reassuring or terrifying depending on how recently you've slept.
Popcorn 🍿, coffee ☕, UPS 🔌, laminated card 🪪, chart-source bookmark 🔖, a domain 🌐, a DHCP reservation 📌, and now a second laminated card that just says compare the renewTimes.
