I Wrote a Self-Healing Script That Could Strand the Machines It Healed

On August 1st I changed three fields on a router at a site I was not standing in, and deliberately made ten machines unreachable. Seven came back on their own in about six minutes. Three did not.

My first conclusion was hardware — a switch port, a dead NIC, something physical I would have to go and touch. That conclusion was wrong. The three machines were stranded by the script I had written to prevent exactly that. And the seven that "recovered" were in worse shape than the three that hadn't.

One address, two machines

Two sites, both on 192.168.1.0/24. An IP address in that range does not designate a machine — it designates a machine and a place you're standing. Measured on 2026-08-01:

remote site : nomad-worker-595b42        →  192.168.1.211
main site   : a Proxmox guest            →  192.168.1.211

That ambiguity had already produced three separate failures. Two of them landed inside Nomad, in two places nothing in its documentation connects, because Nomad carries three notions of "address" that look alike and do not behave alike. The third was quieter and never showed up as an outage at all.

July 29 — lost Nomad quorum, jobs stopped. I changed advertise.serf on a live server. In serf/memberlist the address is the identity: the other members refused the update and repeated Conflicting address for nc-srv-3.global. Mine: X Theirs: Y. Raft lost its leader. Putting the old value back doesn't fix it — the stale entry survives in the other servers' memberlist and in their serf snapshot. I had to stop all three simultaneously and purge /opt/nomad/server/serf/snapshot, with no live node left to re-propagate the entry.

August 1 — the ten remote nodes invisible to Prometheus. The advertise block only covers the agent itself. Services and allocations register with the address of the interface Nomad fingerprints — by default the first routable one, here enp3s0f0, the LAN. So the services were published as 192.168.1.93:9100, an address that from the other site is a different machine, or nothing at all. Prometheus got no route to host on all ten.

And an inventory that lied. A Proxmox VM at the main site ended up listed as a worker at the remote site, because the address matched. A discovery script that sweeps a /24 is only correct from one of the two sites, and nothing in its output tells you which.

Three failures, one root cause, and it is a stupid one: two sites on the same subnet. The overlay network was papering over it. That works — both sites sit behind CGNAT and there is no other way for them to reach each other — but a workaround that has to be right in three different config keys is not a fix.

Renumber the cheap side

The correct fix is not to work around the collision. It's to renumber one of the two sides. Which one is not a judgment call:

The remote machines are already addressed by their overlay address in the Ansible inventory. Their LAN IP is referenced nowhere. They can change address without a single line of configuration moving. Exactly one file in the repo mentioned the old prefix: the default argument of provisioning/discover-minis.sh (192.168.1 → 192.168.2).

The change itself is three fields on the router's LAN page: gateway 192.168.1.1 → 192.168.2.1, DHCP pool start 192.168.1.2 → 192.168.2.2, pool end 192.168.1.254 → 192.168.2.254. The mask stays 255.255.255.0. Rollback is the same three fields in reverse, and since nothing else was ever modified, there is nothing else to undo.

The part that needed a net

Changing the subnet of a site you are not physically at has a specific failure mode. Every machine keeps an address and a default gateway that no longer exist, until its DHCP lease expires. The lease here is 1 day.

No gateway means no internet. No internet means no overlay. No overlay means no SSH. Ten machines on the other side of the city, no remote console, nobody on site to power-cycle them. Up to 24 hours blind.

The runbook's manual escape hatch is a forced renewal pushed over Ansible — which travels over the overlay, which is precisely the thing you lose. That command works for machines that are already fine. It is useless for the ones that need it. The machines had to fix themselves.

The net

A shell script, a systemd unit, and a timer that fires every 2 minutes. If the default gateway fails to answer ping three times in a row, force the DHCP client to re-acquire. Failure count kept in /run/net-selfheal.fails, so it doesn't survive a reboot. Conservative by construction: it does nothing at all while the gateway answers.

[Timer]
OnBootSec=2min
OnUnitActiveSec=2min
AccuracySec=10s

The subtlety that decides whether any of this works: a plain renew is useless here. A renew (or a rebind that finds a server) sends a unicast REQUEST to the DHCP server it already knows, on the subnet that no longer exists. Nobody answers, because there is no longer anybody there. What the machine needs to send is a broadcast DISCOVER — "is there any DHCP server on this wire at all?"

dhcpcd -n does the right thing: it asks for a renewal and falls back to broadcast DISCOVER on its own when nothing replies. That is the entire trick, and it is the only reason the fleet had any chance of coming back.

What actually happened

I deployed the net first, changed the router, and waited. Seven of ten machines came back in roughly six minutes — the three failed checks two minutes apart the design calls for, plus the rebind and the settle. That is the mechanism working exactly to spec: the playbook's own header predicts recovery in about six minutes.

Three stayed dark. Well past the point where they should have healed, and past the point where I could still tell myself they were slow. I wrote them off as a physical problem: bad port, bad cable, a mini that had wedged on boot. I was already planning the trip.

Bug 1: the net could cut the rope

The first version of the script did this:

dhcpcd -k "$IFACE"     # release the lease AND stop the client
dhcpcd    "$IFACE"     # start a fresh one

-k releases the lease and kills the client. If the second command fails — for any reason, at any point — the machine is left with no address and no client to ask for one. Permanently unreachable, achieved by the mechanism whose only job was to prevent that.

This is not a bug of carelessness that a more careful version of me would have avoided. It's a structural property: any two-step where the machine is bare between step one and step two can strand it. The fix is not to be careful across the gap, it's to not have a gap. Never kill the client. Push it with -n, which never gives up the process, and start one only if none is running.

Bug 2: a foreground client inside a oneshot

The second bug is the one I find harder to forgive, because it lives entirely in the seam between two things I wrote.

The script launched dhcpcd without -b. Without that flag, dhcpcd stays in the foreground until it obtains a lease. It was invoked from a Type=oneshot systemd unit, whose TimeoutStartSec defaults to 90 seconds. Past that deadline, systemd kills the whole cgroup — including the dhcpcd that was in the middle of negotiating.

Two minutes later the timer fires again. Same script, same foreground client, same execution killed at 90 seconds. A loop that could never complete, because the thing that would have completed it was being killed on a schedule.

Nothing in the script is wrong when you read the script. Nothing in the unit is wrong when you read the unit. The bug is the interaction, and TimeoutStartSec is a default you inherit whether or not you have read the man page.

The tell

What cracked it was not the three dead machines. It was the seven live ones.

$ pgrep -x dhcpcd
# (nothing — on all seven)

Zero DHCP clients running. Seven machines with a valid 192.168.2.x address, a working gateway, a working overlay, running their workloads — and no DHCP client at all. They had obtained a lease inside the brief window before systemd killed the client, and they kept it, because a lease outlives the process that acquired it.

Which means: at lease expiry, up to 24 hours later, all seven would have dropped off the network at the same time, with nothing left running to ask for a new address. The recovery was an illusion with a 24-hour fuse. The three machines that never came back were the honest ones — same bug, worse timing.

The fix that made the seven genuinely healthy is the same fix that makes the mechanism safe: adopt the lease that already exists instead of releasing it, and make sure a client is running.

The hardened version

Two changes. First, a permanent guard that runs on every tick, unconditionally, before the gateway check — a DHCP client must always be running, even when everything is fine. That is precisely what was missing after the first version:

if ! pgrep -x dhcpcd >/dev/null 2>&1; then
    logger -t net-selfheal "no DHCP client running — starting on $IFACE"
    dhcpcd -b "$IFACE" >/dev/null 2>&1
fi

Second, the escalation path never kills anything:

if [ -z "$GW" ] || ! ping -c 2 -W 2 "$GW" >/dev/null 2>&1; then
    fails=$((fails + 1))
    echo "$fails" > "$STATE"

    if [ "$fails" -ge "$MAXFAILS" ]; then
        # -n = rebind: asks for a renewal, falls back to broadcast DISCOVER
        # on its own when the server does not answer. The client stays alive
        # whatever happens.
        dhcpcd -n "$IFACE" >/dev/null 2>&1
        sleep 10
        logger -t net-selfheal "after rebind: ip=... gw=..."
        echo 0 > "$STATE"
    fi
else
    echo 0 > "$STATE"
fi

The interface is picked by excluding the ones that are never the uplink (lo, docker*, veth*, the overlay interface, bridges), and if none is found the script logs and exits without touching anything.

The three stragglers came back on their own once the corrected script was in place — which is the part that stings. They were never a hardware fault. They were three machines executing my bug faithfully: released lease, no client left running, nothing able to retry. Repair the retry path and they rejoined without anyone touching them. The failure I had attributed to a switch was the safety net, doing exactly what I had written it to do.

The field trick worth keeping

As long as one machine on site has already made the jump, the site is not fully dark. Both subnets can live on the same wire: a temporary secondary address on the old plan is enough to reach a straggler directly from a machine that already moved.

# on a machine already on the new subnet, IFACE = its LAN interface
ip addr add 192.168.1.50/24 dev "$IFACE"

# the straggler still holds its old address — reach it and force a rebind
ssh root@192.168.1.132 "dhcpcd -n $IFACE"

# clean up
ip addr del 192.168.1.50/24 dev "$IFACE"

Neither ARP nor the kernel's routing table objects to two /24s being on-link on the same interface. It only requires that one machine on site made it across — and after a renumbering, some always do.

In this incident it recovered nothing, and that is worth saying plainly: the three machines I was hunting were not sitting on the old subnet waiting to be poked, they had no address and no client at all. A tool that reaches the old plan cannot help a machine that is no longer on any plan. I keep it staged anyway, because "stuck on the old addressing" is the failure this operation is actually supposed to produce — and next time the stragglers may really be stragglers rather than casualties of my own script.

The acceptance check

The runbook names its pass criteria up front, so that "it seems fine" never gets to be the verdict. Same numbers before the change and after — here is where they landed once the corrected script had been through the fleet:

The list as written was missing the check that mattered most, so I added it and verified it separately: a DHCP client is actually running on every machine — ten out of ten, once the fixed script had adopted each existing lease. Reachability was already in the criteria, and reachability is exactly the thing that lied. An IP address designates one machine again, from either side of the city. The discovery script is trustworthy from any location. The whole class of bugs I'd been chasing across three config keys is gone at the root, and the overlay is back to being a bridge between sites instead of a patch over an addressing mistake.

Two things I'm taking with me

A self-healing mechanism that manipulates the network must be structurally incapable of leaving the machine without a means to retry. Not "careful about it" — incapable. Release-then-restart is a sequence with a hole in the middle; rebind is a single step that never surrenders the client. And the happy path is not the path that matters: my version worked perfectly every time the restart succeeded, which on my desk is every time. The branch I never exercised is the branch that ran in production, on machines I couldn't reach.

"It recovered" is not the same as "it is healthy." Seven machines answered ping, held a valid address, and ran their jobs, while carrying a 24-hour fuse. The check that mattered was never "does it respond now" — it was "is the thing that lets it respond tomorrow still running?" Verify the mechanism, not the symptom. The symptom is what the mechanism was supposed to hide.

The script is now the smallest amount of automation I could write that cannot make things worse. I only know that because the previous one did.