Hey all 👋
Episode 11 ended with a sentence I had been dodging for six episodes: the actual fix is a cable.
Quick recap of why. In Episode 5, joining the Dell moved dqlite off loopback and onto msi's Wi-Fi address, and a power cut showed what happens when that address disappears. In Episode 11, Wi-Fi power save made it disappear all by itself: the datastore cancelled its own queries, the watch cache froze, and the cluster spent thirty-eight hours looking healthy while doing nothing. Two incidents, one root cause. The most important process in the cluster lived on the least reliable link in the house.
So this episode: the cable. And a cable needs something on the other end, which means the Raspberry Pi finally leaves the drawer. The same Pi that has been joining the lab "next episode" since Episode 3. Episode 4: still in the drawer. Episode 5: "thriving, unbothered." Episode 9: "next episode, the drawer opens. Probably." Episode 10: "The drawer has heard promises before."
The drawer opened. I budgeted one evening. It was one evening, technically, if an evening is allowed to end at 4:13 AM.
🗺️ The Plan
The Archer stays the house's router. The Pi becomes the lab's: its own subnet, its own DHCP, its own DNS and its own firewall, all in config files I can read. The Pi joins the house over Wi-Fi, the lab hangs off its Ethernet port through the switch, and the Pi routes and NATs between the two.
The important part is what this does to the cluster. msi and dell talk to each other through the switch, on gigabit copper, and that traffic never touches the Pi or the Wi-Fi. The datastore binds to an address on a cable. The Pi only carries the lab's internet traffic, so if it falls over, the lab loses the internet and the cluster keeps its memory.
The address plan, decided before touching anything, because Episode 10 taught me that addresses get memorized in places I forget about:
| Range | What lives there |
|---|---|
10.10.0.x | Network gear: Pi .1, switch .2 |
10.10.1.x | Kubernetes nodes, static: msi .10, dell .11 |
10.10.2.x | Reserved for the NodeMCU and future IoT experiments |
10.10.100.x | DHCP pool, for anything I did not plan for |
The first draft was a /24. I widened it to a /16, so 10.10.anything means "the lab" and every role gets its own neighbourhood. Then I checked it against MicroK8s's own ranges (pods on 10.1.0.0/16, services on 10.152.183.0/24) before committing. No overlap.
Two rules came with the plan:
- Nodes carry their own static addresses. The Pi's DHCP is only for strangers. If the Pi reboots, nobody's address moves, and nothing from Episode 10's census has to be redone.
- The house can see into the lab, by real address. The Archer gets a static route for
10.10.0.0/16via the Pi, and the Pi does not NAT traffic headed for the house. From my workstation, msi is10.10.1.10, not "something behind the Pi." kubectl, certificates and logs all see the node's actual IP.
🍓 Sidequest 1: The Pi That Was Not There
Raspberry Pi OS, flashed with Imager's customisation screen filled in: hostname gw, a user, SSH, Wi-Fi credentials. Plugged it in, opened the Archer's client list, waited for a new device.
Nothing. Waited longer. Still nothing.
Without a network, a Pi has exactly two ways to talk to you: a red LED and a green one. Red was steady, so power was fine. Green was blinking, but not the irregular flicker of disk activity. An even, metronome blink, forever. So I did what any engineer would do and timed it with a stopwatch:
52 blinks in 36.24 seconds. No pauses. Every blink the same length. About 1.4 blinks a second. The Pi's boot error codes are groups of long and short flashes with a pause between repeats, so you can count them. This had no pauses and no long flashes, and matched nothing in the documentation. Diagnosing a computer by timing its LED with a stopwatch is a new low for this series, and this series has a laminated card.
Then I plugged in a monitor:
gw login: Booted, healthy, hostname applied. The LED was a red herring. Green, technically.
Which left me with a login prompt and no keyboard, a very specific kind of torture. The way in was a cable from my workstation straight into the Pi. Neither end of a direct cable runs DHCP, so I made my workstation one:
sudo nmcli con add type ethernet ifname <wired-iface> con-name pi-direct ipv4.method shared sudo nmcli con up pi-direct NetworkManager's shared mode turns a port into a tiny router: it takes 10.42.0.1, runs a DHCP server and NATs anything behind it. The Pi asked for an address, got one, and I was in.
The Wi-Fi problem turned out to be simple. The SSID I had given the Pi is a separate IoT network on the Archer, broadcasting on 5 GHz only, and the Pi's scan only listed 2.4 GHz networks. I turned on 2.4 GHz for that SSID and the Pi joined in seconds. Twenty minutes, start to finish.
I would like to frame that moment, because it was the last time anything that night took twenty minutes.
📡 Sidequest 2: The Channel That Was Not There
Then I got greedy and wanted 5 GHz. Not for bragging rights. The Pi's Wi-Fi is now the lab's only way out, and from where the Pi sits, six networks were shouting over each other on 2.4 GHz channel 6, mine included. On 5 GHz, my network showed up on channel 149 with a stronger signal than its 2.4 GHz twin:
$ nmcli -f SSID,CHAN,SIGNAL,SECURITY dev wifi list | grep -i serve serve 149 80 WPA2 serve 6 67 WPA2 Better signal, less noise, more bandwidth. An obvious upgrade. Five hypotheses later, every one of them dead, it was 3 AM.
🔬 Hypothesis 1: The Pi Does Not Do 5 GHz
Plausible, since it had just failed to see my 5 GHz-only network.
The test: ask the kernel what the radio is allowed to use.
$ iw reg get global country US: DFS-FCC ... phy#0 country 99: DFS-UNSET (2402 - 2482 @ 40), (6, 20), (N/A) (2474 - 2494 @ 20), (6, 20), (N/A) (5140 - 5360 @ 160), (6, 20), (N/A) (5460 - 5860 @ 160), (6, 20), (N/A) Two findings. The regulatory domain said US, not IN, so the country setting had not applied. And the phy#0 block, the Wi-Fi chip's own rules, covers every 5 GHz channel the Archer can use, 149 included. The scans agreed: neighbours on channels 36 to 48, my network on 149.
Dead. The Pi does 5 GHz fine.
So I went to the Archer and turned off 2.4 GHz for that SSID, expecting the Pi to roll over to 5 GHz. It rolled over and died. Disconnected, never came back. 2.4 GHz back on, Pi back online, me back to square one, with a sudden interest in rollback plans.
🧅 Interlude: Three Layers of Config for One Wi-Fi Password
Before changing anything else, I wanted to know who owned the Wi-Fi config. nmcli answered with a name I did not pick:
$ nmcli -f NAME,TYPE,DEVICE con show NAME TYPE DEVICE netplan-wlan0-serve wifi wlan0 lo loopback lo netplan-eth0 ethernet -- netplan-. On this Pi OS release, Imager's settings are applied by cloud-init, which writes netplan YAML, which generates NetworkManager profiles. And it loops back: /etc/netplan/ held two 90-NM-<uuid>.yaml files, which is NetworkManager writing every nmcli edit back into netplan. Three tools, one Wi-Fi password, and a cycle.
The one that worried me was cloud-init, which can render the network config again later and quietly undo everything. So it got told to stop:
echo 'network: {config: disabled}' | sudo tee /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg After that, netplan is the source of truth, and nmcli is a perfectly good way to edit it. (netplan also spent the whole night warning me that a file it ships itself has permissions that are too open, which is the most Linux sentence I have ever read.)
🔬 Hypothesis 2: NetworkManager Is Not Trying Hard Enough
First attempt: pin the Wi-Fi profile to 5 GHz in the YAML and apply it. Any change that can cut off the SSH session making it gets a dead man's switch, so this ran as its own systemd unit, which outlives the session if the Wi-Fi drops:
sudo systemd-run --unit=netplan-test bash -c ' netplan apply; sleep 30 if ping -c3 -I wlan0 192.168.0.1; then echo OK; exit 0; fi echo "failed, rolling back" rm -f /etc/netplan/*.yaml; cp -a /root/netplan.bak/. /etc/netplan/; netplan apply' It rolled back:
PING 192.168.0.1 (192.168.0.1) from 10.10.0.1 wlan0: 56(84) bytes of data. 3 packets transmitted, 0 received, 100% packet loss, time 2036ms failed, rolling back from 10.10.0.1: ping borrowed the Ethernet port's address, because wlan0 did not have one. NetworkManager's log for that window was more interesting for what it did not say. netplan apply had restarted NetworkManager, wlan0 came back as disconnected, and then nothing. Not a failed attempt. No attempt. NetworkManager only auto-connects to networks it has seen in a scan, so maybe thirty seconds after a restart was just an impatient test.
The test: skip autoconnect, force it, and give it a longer fuse.
nmcli con mod netplan-wlan0-serve 802-11-wireless.band a nmcli -w 60 con up netplan-wlan0-serve Error: Connection activation failed: The Wi-Fi network could not be found ... <warn> device (wlan0): Activation: (wifi) association took too long, failing activation <info> device (wlan0): state change: config -> failed (reason 'ssid-not-found') It scanned for twenty-two seconds and found nothing. The rollback unpinned the band, and the Pi rejoined on 2.4 GHz in three seconds.
Dead. Not impatience. It could not see the network.
🔬 Hypothesis 3: The Country Code Ate Channel 149
Somewhere in the middle of all this, I had fixed that US with raspi-config nonint do_wifi_country IN. And now a fresh scan showed nothing on channel 149 at all. Not my network, not the neighbours'. Channels 36 to 48 were busy (including, I am pleased to report, a neighbour called Stairway to LAN), and 149 was empty. Earlier that night, 149 had been right there.
The theory: the Wi-Fi chip's firmware has its own table for IN, and it blocks the upper band.
The test: ask the kernel, then put the country back.
$ iw list | grep -E '5745|5180' * 5180.0 MHz [36] (20.0 dBm) * 5745.0 MHz [149] (20.0 dBm) Both enabled. Then sudo iw reg set US and a rescan: still nothing on 149.
Dead, at least from anywhere I could reach.
🔬 Hypothesis 4: The Router Stopped Broadcasting
Maybe all that toggling had knocked the Archer's 5 GHz radio over.
The test: ask msi, which lives on the same network.
msi@msi:~$ iw dev Interface wlo1 ssid serve type managed channel 149 (5745 MHz), width: 80 MHz, center1: 5775 MHz Dead. The network was right there on 149. One laptop could see it. The Pi could not.
🔬 Hypothesis 5: Band Steering (Confidently Wrong)
Out of hypotheses, I started changing variables. I pinned the Archer's 5 GHz to channel 48, a plain non-DFS channel every regulatory table allows, and split 2.4 GHz off into its own SSID. The Pi joined 5 GHz straight away. Then, for science, I moved the Archer back to 149. The Pi stayed connected.
So, new theory: with one SSID on both bands, the Archer had been steering the Pi onto 2.4 GHz, and splitting the SSIDs fixed it. Channel 48 was a coincidence. I believed this for about three minutes.
The test: reboot the Pi, with the router still on 149.
Nov 19 03:04:17 gw NetworkManager[602]: <info> device (wlan0): state change: unavailable -> disconnected (reason 'supplicant-available') ... three minutes of nothing ... Nov 19 03:07:36 gw NetworkManager[602]: <info> policy: auto-activating connection 'netplan-wlan0-serve' Nov 19 03:07:37 gw NetworkManager[602]: <info> device (wlan0): Activation: (wifi) Stage 2 of 5 (Device Configure) successful. Connected to wireless network "serve" Three minutes with no attempts and no errors. 03:07:36 is the moment I moved the Archer back to channel 48.
Dead. And the reboot had just demonstrated the only explanation left.
🎯 The Answer: Finding vs Staying
Joining a network and staying on it are two different paths.
- Joining starts with a scan. The client has to hear the access point on its channel, then associate. On channel 149, the Pi never got through that, not once all night. Early on it would sometimes list my network there; by the end it could not see the channel at all.
- Staying does not need a scan. When an access point changes channel, it warns its clients first with a Channel Switch Announcement in its beacons, and every client already associated just follows it there.
So the Pi could follow the Archer to 149. It could never find the Archer on 149. Every time the Pi had to find the Archer on 149, it failed. Every time the Archer was on 48, it worked. "Works until reboot" was the signature the whole time.
I never found out exactly why the Pi cannot get onto 149 with this router. My best guess is the Wi-Fi chip's firmware, which keeps its own opinion about that channel, separate from what iw list reports. Episode 11 had a sentence for this situation: I found the fix, which was good enough at 3 AM. It was 3 AM again.
The fix:
- 5 GHz pinned to channel 48 on the Archer. No more Auto. Auto channel on a router with an unattended gateway as a client is a reboot-shaped time bomb, and a power cut, Episode 5's favourite event, reboots everything at once.
- 2.4 GHz off. Nothing in the lab needs it.
- Power save off on the Pi's Wi-Fi, because Episode 11:
sudo nmcli con mod netplan-wlan0-serve 802-11-wireless.band a 802-11-wireless.powersave 2 With the Archer on 48, the Pi joined within a second of the switch.
🧱 Building the Router
With an uplink that survives a reboot, the thing I actually came to build took a fraction of the time the Wi-Fi did. Three pieces: forwarding, a firewall with NAT, and DHCP plus DNS.
The Ethernet side gets a static address and no gateway, because the Pi is the lab's gateway:
sudo nmcli con mod netplan-eth0 ipv4.method manual ipv4.addresses 10.10.0.1/16 ipv6.method disabled sudo nmcli con up netplan-eth0 echo 'net.ipv4.ip_forward=1' | sudo tee /etc/sysctl.d/99-router.conf sudo sysctl --system The firewall
/etc/nftables.conf:
#!/usr/sbin/nft -f flush ruleset define LAN = "eth0" define WAN = "wlan0" define LAB = 10.10.0.0/16 define HOME = 192.168.0.0/24 table inet filter { chain input { type filter hook input priority 0; policy drop; ct state established,related accept ct state invalid drop iif lo accept ip protocol icmp accept meta l4proto ipv6-icmp accept tcp dport 22 accept udp dport 5353 accept iifname $LAN udp dport { 53, 67 } accept iifname $LAN tcp dport 53 accept } chain forward { type filter hook forward priority 0; policy drop; ct state established,related accept ct state invalid drop iifname $LAN oifname $WAN accept iifname $WAN oifname $LAN ip saddr $HOME accept } } table ip nat { chain postrouting { type nat hook postrouting priority srcnat; policy accept; oifname $WAN ip saddr $LAB ip daddr != $HOME masquerade } } The line that matters most is the last one. The lab gets NAT on its way to the internet, but not on its way to the house. Episode 10 found seven places where my cluster had memorized addresses, and I was not going to add an eighth that lies about them.
The rest is boring on purpose. Input is closed except SSH, mDNS, ICMP (ICMPv6 too, without which the drop policy quietly breaks IPv6 neighbour discovery), and DHCP and DNS from the lab side only. Forwarding lets the lab out and the house in, and nothing else.
Loading a firewall over SSH is how people end up driving to the datacenter at midnight. My datacenter is a shelf, but the principle stands. So, a dead man's switch: a timer that flushes the rules in two minutes unless I prove I can still get in.
sudo nft -c -f /etc/nftables.conf # syntax check sudo systemd-run --on-active=120 --unit=nft-rollback nft flush ruleset sudo nft -f /etc/nftables.conf # open a NEW ssh session; if that works: sudo systemctl stop nft-rollback.timer sudo systemctl enable --now nftables It is netplan try, but for firewalls. Episode 3 said netplan try deserves a fan club. This is its cousin.
DHCP and DNS
/etc/dnsmasq.d/lab.conf:
interface=eth0 bind-dynamic domain-needed bogus-priv no-resolv server=1.1.1.1 server=8.8.8.8 domain=lab local=/lab/ dhcp-range=10.10.100.1,10.10.100.254,255.255.0.0,12h dhcp-option=option:router,10.10.0.1 dhcp-option=option:dns-server,10.10.0.1 host-record=gw.lab,10.10.0.1 host-record=sw.lab,10.10.0.2 host-record=msi.lab,10.10.1.10 host-record=dell.lab,10.10.1.11 DHCP only hands out the 10.10.100.x pool. The static machines get names through host-record, so msi.lab resolves without msi ever asking the Pi for anything.
The house learns where the lab is
On the Archer, Advanced > Network > Routing: a static route for 10.10.0.0, mask 255.255.0.0, gateway 192.168.0.203. That is the Pi, which has a DHCP reservation of its own, because Episode 10. Now every device in the house can reach 10.10.anything without a single route configured on the device itself.
🔌 Sidequest 3: The Laptop That Never Asked
Cable into msi, other end into the switch. msi's address on that port: none.
msi@msi:~$ networkctl IDX LINK TYPE OPERATIONAL SETUP 1 lo loopback carrier unmanaged 2 enp3s0 ether degraded configuring 3 wlo1 wlan routable configured 4 cali8b2161aee6b ether degraded unmanaged ... 12 vxlan.calico vxlan routable unmanaged ... 37 links listed. Thirty-seven links, and thirty-four of them are Calico's: thirty-three pod veths and a VXLAN tunnel. Of the three left, one was the cable I cared about, sitting at degraded / configuring. Link up, no address, networkd waiting.
Waiting for instructions it was never given. msi's netplan was written by the Ubuntu installer, and the Ethernet port's entry had a name and nothing else:
ethernets: enp3s0: match: macaddress: <msi's MAC> set-name: enp3s0 No dhcp4, no addresses. The installer named the port and walked away. I rebooted msi to see if that helped. It did not, because a reboot faithfully reloads the same absence of instructions.
msi does not want DHCP anyway. So the fix was staged: give the port its permanent address now, but no gateway, so the default route, and with it the cluster, stays on Wi-Fi until the real cutover:
ethernets: enp3s0: match: macaddress: <msi's MAC> set-name: enp3s0 dhcp4: false addresses: [10.10.1.10/16] optional: true Applied with sudo netplan try (fan club still active). optional: true keeps boot from waiting on a cable that might not be there.
Then my favourite test of the night. I wanted to prove the whole new path, switch to Pi to NAT to Wi-Fi to the internet, without moving msi's default route and risking the cluster. So I moved exactly one address:
ping -c3 10.10.0.1 # the Pi, over the switch dig +short dell.lab @10.10.0.1 # dnsmasq answering sudo ip route add 1.1.1.1/32 via 10.10.0.1 dev enp3s0 ping -c3 1.1.1.1 # the internet, via the Pi tracepath -n 1.1.1.1 | head -3 # first hop: 10.10.0.1 sudo ip route del 1.1.1.1/32 via 10.10.0.1 A /32 host route sends one destination through the Pi. Everything else, dqlite included, carried on over Wi-Fi, none the wiser.
🪞 Sidequest 4: The Ping That Came Back the Wrong Way
msi now had a leg in each network, so I tried SSH over the new one:
╭─sunny@sunny ~ ╰─$ ping -c3 10.10.1.10 64 bytes from 10.10.1.10: icmp_seq=1 ttl=64 time=9.38 ms 64 bytes from 10.10.1.10: icmp_seq=2 ttl=64 time=8.11 ms 64 bytes from 10.10.1.10: icmp_seq=3 ttl=64 time=5.65 ms ╭─sunny@sunny ~ ╰─$ ssh msi@10.10.1.10 kex_exchange_identification: read: Connection reset by peer Connection reset by 10.10.1.10 port 22 Ping works, SSH does not. Episode 3 already taught me not to trust a ping that works, and this one hid its confession in the TTL.
Linux sends packets with a TTL of 64, and every router that forwards one takes one off. My request went workstation -> Archer -> Pi -> msi. A reply coming back through the Pi would have lost at least one and arrived as 63 or lower. It arrived as 64. It never went through a router at all.
What actually happened: msi got a packet from 192.168.0.x, noticed it had a Wi-Fi interface sitting directly on 192.168.0.x, and replied straight over the air. Requests went in through the Pi; replies came back over Wi-Fi. Ping does not care which way its reply travels. TCP does, or rather the stateful boxes along the way do: they only ever saw half of each conversation, and a connection missing half its packets does not survive the handshake. I did not stop to work out which box sent the reset. I stopped sending packets the wrong way instead.
The asymmetry fixes itself the moment msi's Wi-Fi goes away, which was the whole plan anyway. Until then, the Pi became a jump host. The Pi and msi sit on the same switch, so both directions stay on the wire:
ssh -J rpi@192.168.0.203 msi@10.10.1.10 Or permanently, in ~/.ssh/config:
Host msi HostName 10.10.1.10 User msi ProxyJump rpi@192.168.0.203 That mattered twice, because removing msi's Wi-Fi was the next step. Episode 3 warned about being one YAML indent away from locking yourself out over the very connection you are using. This was a session that removing Wi-Fi could not cut off.
✂️ The Cutover (Episode 10, Abridged)
Episode 10 was an RCA of seven places where my cluster had memorized its own address. This time I had the census before I started, which turns an RCA into a checklist. Short version here, because Episode 10 has the long one.
First, look before touching:
$ sudo cat /var/snap/microk8s/current/var/kubernetes/backend/cluster.yaml - ID: 3297041220608546238 Address: 192.168.0.201:19001 Role: 0 One member, a voter. The Dell is not in there at all, because it joined as a worker. dqlite is a party of one, so there is no quorum to negotiate. Backups first (the backend directory, csr.conf.template, the args folder), because paranoia is a feature.
The checklist:
- Calico. Instead of Episode 10's
can-reach=<an IP that will one day stop existing>, a rule by subnet:IP_AUTODETECTION_METHOD=cidr=10.10.0.0/16. It works for both nodes, whatever their interfaces are called. The rollout stalled at "1 out of 2 new pods have been updated," because pod two belongs to the Dell, which is resting. - dqlite. Swap the address in
info.yamlandcluster.yaml, then reconfigure with the bundled tool. It answeredOK. - Certificates.
IP.10 = 10.10.1.10incsr.conf.template, then regenerate the server cert. - Pin the addresses.
--node-ipfor the kubelet,--advertise-addressfor the API server, so nothing auto-detects its way back to Wi-Fi. - msi's netplan. The
wifisblock deleted entirely; default route and DNS moved to the Pi. - Workstation kubeconfig. One
sed.
For the record:
sudo microk8s kubectl -n kube-system set env ds/calico-node IP_AUTODETECTION_METHOD=cidr=10.10.0.0/16 sudo microk8s stop B=/var/snap/microk8s/current/var/kubernetes/backend sudo sed -i 's/192.168.0.201:19001/10.10.1.10:19001/' $B/info.yaml $B/cluster.yaml sudo /snap/microk8s/current/bin/dqlite \ -s file://$B/cluster.yaml -c $B/cluster.crt -k $B/cluster.key -f json \ k8s ".reconfigure $B/ $B/cluster.yaml" sudo sed -i '/^IP.9 = /a IP.10 = 10.10.1.10' /var/snap/microk8s/current/certs/csr.conf.template echo '--node-ip=10.10.1.10' | sudo tee -a /var/snap/microk8s/current/args/kubelet echo '--advertise-address=10.10.1.10' | sudo tee -a /var/snap/microk8s/current/args/kube-apiserver sudo microk8s start sudo microk8s refresh-certs --cert server.crt One confession. After step 3, I checked the certificate, found no new IP in it, and was briefly alarmed, until I noticed I had skipped the two lines that start MicroK8s and regenerate the certificate. A certificate check only tells the truth after you have made the certificate.
msi's final netplan:
network: version: 2 ethernets: enp3s0: match: macaddress: <msi's MAC> set-name: enp3s0 dhcp4: false addresses: [10.10.1.10/16] routes: - to: default via: 10.10.0.1 nameservers: addresses: [10.10.0.1] search: [lab] sudo netplan try, from the jump-host session. Wi-Fi gone. Enter pressed. Then, on the workstation:
sed -i 's#https://192.168.0.201:16443#https://10.10.1.10:16443#' ~/.kube/config msi came back Ready on 10.10.1.10, its only address, on a cable. The datastore that spent Episodes 5 through 11 as a hostage of a Wi-Fi card is now bound to a port with a wire in it.
╭─sunny@sunny ~ ╰─$ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME dell NotReady <none> 81d v1.35.6 192.168.0.202 <none> Ubuntu 26.04 LTS 7.0.0-30-generic containerd://2.1.6 msi Ready <none> 81d v1.35.6 10.10.1.10 <none> Ubuntu 26.04 LTS 7.0.0-43-generic containerd://2.1.6 🧠 What This Taught Me
- Joining a network and staying on it are different paths. Joining starts with a scan; following a channel change uses the access point's Channel Switch Announcement and skips the scan. "Works until reboot" is the signature of a discovery problem.
- Pin the Wi-Fi channel for anything that has to come back on its own. Auto channel plus an unattended Wi-Fi uplink is a reboot-shaped time bomb, and power cuts reboot everything at once, at night, while nobody is watching.
- The TTL is a free traceroute. A reply that should have crossed a router but arrives with 64 never crossed one. Ping working, TCP failing, and a TTL that is too high add up to asymmetric routing.
- A host with a leg in two networks answers from the leg closest to the sender. Staged cutovers create exactly that. Reach the host through the new network (a jump host on it works) until the old leg is gone.
-
degraded / configuringmeans the cable is fine and the config is empty. A linked interface with no instructions waits forever, silently, and a reboot reloads the same nothing. - On this Pi OS, network config is three layers deep. cloud-init writes netplan, netplan generates NetworkManager profiles, and NetworkManager writes
nmcliedits back into netplan. Turn off cloud-init's network module, then edit with either of the other two. - Every change that can cut off your session gets a timer that undoes it.
netplan try, asystemd-run --on-activeflush for nftables, a rollback unit for the Wi-Fi. Run them as systemd units, so they outlive the SSH session they protect. - Test one route, not the default route. A
/32through the new gateway proves the whole path while everything else stays where it is. - An LED is a log with one bit. It can tell you there is power and that something is happening. A monitor tells you what.
🚀 What Is Next
The Dell is still on its holiday, and it has an Episode 11 backlog waiting: ghost pods stuck in Terminating and a triggerer StatefulSet that will not replace them. When it wakes up, it gets the same treatment as msi: a static 10.10.1.11, its own --node-ip, and Episode 10's census item two, the worker's provider.yaml, pointed at 10.10.1.10. Calico is already done for it, because the subnet rule covers both nodes.
After that, the NodeMCU finally gets a home in 10.10.2.x. And the third-voter-shaped hole in the cluster's HA story is still there. The Pi was supposed to fill it, back when it lived in a drawer. Instead it got a job as the lab's router, which, given this series, might be the more important one.
💬 Final Thoughts
Almost four hours of this night went to Wi-Fi. Less than one went to everything I actually came to build. The router config was easy, the firewall was easy, and the cutover was a checklist. What ate the night was a Raspberry Pi that could not find one specific radio channel, which a laptop on the same network was using the entire time.
But the datastore lives on a cable now. The Wi-Fi card can nap and the router can fall over, and the cluster keeps talking to itself through a switch that has never once had an opinion. Episode 9 demoted the router from front door to hallway. This episode demotes it again: it is now just the lab's internet provider, and a cluster does not need its internet provider to remember its own state.
The Pi left the drawer, became a router, and spent its first night being stubborn about a radio channel. Honestly, it fits right in.
Popcorn 🍿, coffee ☕, UPS 🔌, two laminated cards 🪪, chart-source bookmark 🔖, a domain 🌐, a DHCP reservation 📌, and now a Pi 🥧, out of the drawer at last, with a sticky note on it that says channel 48.

