Changing a Live Server's Address Changes Its Identity

I broke a three-server Nomad control plane by changing one line in a config template. Not a wrong line — it held the same address as the two lines above it, which I changed at the same time and which caused no trouble at all. The difference is that in serf/memberlist, one of those three lines is not a setting. It is the node's name.

Causing the outage took a single playbook run. The cleanup took three days, and the cleanup is the part worth reading: three obvious repairs do not work, one of them by moving the fault rather than removing it, and the one that does had to happen on all three servers inside the same window.

The setup

The lab is a Nomad + Consul control plane: three servers (nc-srv-1, nc-srv-2, nc-srv-3) running as Debian LXC guests on a single Proxmox host at the main site, region global, datacenter dc1, Nomad 2.0.4, TLS and ACLs on, everything provisioned by Ansible. Two Nomad clients run the workloads: Prometheus, Grafana, a whoami service, node-exporter, a CI runner.

Ten more machines sit at a second, remote site and want to join as Nomad clients. That is where the whole problem starts, and it is embarrassingly mundane: both sites use 192.168.1.0/24. 192.168.1.211 is a worker at the remote site, and it is also a Proxmox guest at the main site. The same address means two different machines depending on where you are standing. Telling them apart took a different SSH host key, a different OS version and a Proxmox MAC prefix — I had to check, because that ambiguity once pulled a virtual machine into an inventory of physical ones.

The workaround was an overlay network (Tailscale): every node gets one stable address that is unique and means the same thing from either site. So the obvious move was to make the cluster advertise its overlay addresses. That obvious move is what broke it.

The failure: one variable, three addresses

The Ansible template for nomad.hcl had this. Read the third line inside the block.

{% if nomad_network | default(cluster_network | default('lan')) == 'tailscale' %}
{% set adv = hostvars[inventory_hostname].ts_ip %}
{% else %}{% set adv = ansible_host %}{% endif %}
advertise {
  http = "{{ adv }}"
  rpc  = "{{ adv }}"
  serf = "{{ adv }}"        # this is the line that broke the cluster
}

One variable, three addresses, one playbook run across all three live servers. Within seconds the journals filled up with this:

memberlist: Conflicting address for nc-srv-3.global.
  Mine: 192.168.1.223:4648  Theirs: <overlay addr>:4648

Port 4648 is serf — the gossip layer, not Raft, not the RPC. Nomad lost its leader, scheduling stopped, and the cluster spent the next few minutes arguing with itself about who nc-srv-3 was. The rollback was a re-run of the same playbook without the override: configs rewritten at 18:58:25, agents restarted at 18:58:28, :29, :29. A leader came back. I wrote it off as a bad afternoon and moved on.

The scar you cannot see for two days

Two days later I ran a full audit of the cluster, expecting to find nothing interesting about that incident. Instead:

nomad server members  → seen from srv-1 and srv-2: nc-srv-3 = <overlay addr>
                        seen from srv-3 itself   : nc-srv-3 = 192.168.1.223
nomad operator raft   → leader nc-srv-2; peer nc-srv-3 = <overlay addr>:4647
/etc/nomad.d/nomad.hcl on srv-3 → advertise { http/rpc/serf = "192.168.1.223" }
consul members        → nc-srv-3 = 192.168.1.223            (correct)

The rollback had rewritten the file on disk. It had not moved memberlist an inch. The cluster held two contradictory beliefs about the same node, and logged the disagreement roughly once a minute — about 1,450 Conflicting address lines in 24 hours in the journal of nc-srv-2 alone:

nc-srv-2: Conflicting address for nc-srv-3.global.
          Mine: <overlay addr>:4648  Theirs: 192.168.1.223:4648

The mechanism is simple once you accept the premise. In memberlist, whoever claims a name first keeps the address bound to it and rejects corrections. And the state does not only live in RAM: /opt/nomad/server/serf/snapshot — about 1.2 kB, a cache of member addresses — still listed the overlay address as the last known entry for that node. Its mtime was the process start time, unchanged for two days. A naive stop/start re-injects it.

Three consequences, and I had anticipated exactly zero of them:

• The quorum silently depended on the overlay daemon. Raft was routing to a peer that lives on the same physical host, through a WireGuard tunnel, because that is the address it had. If tailscaled stopped inside that container, quorum dropped to 2/3 with no margin. Nothing in the config said so.
• The drift had already reached the data plane. GET /v1/agent/servers on a client returned an overlay address that client had never had in any configuration file. A client's servers list is only a seed: the server heartbeat replaces it, using each peer's advertised RPC address, which is propagated as the serf tag rpc_addr.
• Any further address reconfiguration on that node was a silent no-op. While the conflict stands, the other two servers ignore every tag update coming from the conflicted node. I could edit nomad.hcl, restart the agent, confirm the file on disk was exactly right, and see nothing at all change in nomad server members. This is the consequence that would have cost me a full day of "why does Nomad ignore its own config file".

Three repairs that did not work

1. force-leave on a running node

nomad server force-leave nc-srv-3.global from the other two, agent still up. This does nothing durable: the node is still gossiping, and it re-announces the stale address out of its own serf snapshot within a gossip round — evicted and back before the command output finishes printing.

2. Cleaning the drifted server alone

Better idea: take the node out of the conversation first. Stop Nomad on the drifted server (quorum stays at 2/3, never at risk), run force-leave from the other two while it is absent, back up and empty its serf snapshot, start it again.

It half-worked, which is the worst outcome. The node now saw itself correctly at 192.168.1.223. And immediately started logging:

serf: Node name conflicts with another node at <overlay addr>:4648

The stale entry had not been destroyed. It had moved. The other two servers still held it in their own memberlist and re-seeded it, by gossip, into the node I had just cleaned. I had spent thirty minutes carefully relocating the bug.

3. Cleaning them one at a time, in sequence

This one I did not have to run to the end. As long as one live member holds the entry, gossip pushes it back into whichever node was just purged — the same failure, mirrored. There is no rolling version of this operation — which is the whole lesson compressed into one sentence: a serf address change requires a simultaneous stop, never a rolling one.

What worked: a short simultaneous window

Stop all three, purge all three membership caches while nothing is alive to re-propagate, start all three.

# on all three servers, in parallel — the window opens here
systemctl stop nomad

F=/opt/nomad/server/serf/snapshot
cp "$F" "$F.bak-$(date +%F-%H%M%S)"
: > "$F"                      # membership address cache only
                              # /opt/nomad/server/raft is NOT touched

systemctl start nomad

Measured outcome:

• A leader was back within seconds of the restart, without any manual peer surgery.
• Three voters, zero overlay addresses in nomad operator raft list-peers, and journalctl -u nomad --since "-1min" | grep -c "Conflicting address" → 0.
• Roughly 1 to 2 minutes with no leader: no new scheduling, no nomad job run. No allocation was interrupted — the running containers never stopped, and Consul was not touched at all.

The real fix is a schema change, not a repair

Purging the caches restored the state. It does nothing to stop me from doing it again next month. The actual fix is to stop coupling two things that were only accidentally the same value.

So the template splits them, and deliberately does not offer a switch for the first one:

{# serf = the member's identity in the gossip. NEVER change it on a live server.
   rpc/http = transport: free to change, propagated via the serf tag rpc_addr. #}
{% set serf_adv = ansible_host %}
{% if nomad_advertise_rpc_network | default('lan') == 'tailscale' %}
{%   set rpc_adv = hostvars[inventory_hostname].ts_ip %}
{% else %}
{%   set rpc_adv = ansible_host %}
{% endif %}
advertise {
  http = "{{ rpc_adv }}"
  rpc  = "{{ rpc_adv }}"
  serf = "{{ serf_adv }}"
}

Deployed one server at a time, leader last, waiting for failure tolerance to return to 1 and three voters before moving to the next. Zero conflicts at every step. Afterwards, nomad server members shows Addr in 192.168.1.x — serf, untouched — while rpc_addr is on the overlay. That asymmetry looks like a bug in a dashboard and is exactly the intended result.

The servers became reachable from the remote site without the gossip layer ever noticing, which is what let the 10 remote workers join. Twelve workers ready, Raft 3/3, jobs intact. One line of difference between a broken cluster and a two-site one.

Consul stayed entirely on the local LAN, on purpose. Its LAN gossip is tuned for a sub-10 ms datacenter; the cross-site round-trip here measures 57 / 87 / 322 ms with a mean deviation of 49 ms. Nomad's serf shrugs at that. Consul's would flap the remote nodes in and out of the pool forever. Stretching one Consul datacenter across both sites would have been the exact symmetric version of the mistake I had just made.

A third address, one day later

There is a punchline. The day after the servers went multi-site, Prometheus reported no route to host on all ten remote workers. The advertise block only concerns the agent itself. Services and allocations register under the address of the interface Nomad fingerprints, by default the first routable one — the LAN. So the workers were publishing themselves as 192.168.1.93:9100, an address that from the other site points at a different machine, or at nothing.

client { network_interface = "tailscale0" } fixes it, with a corollary that cost me another twenty minutes: an allocation registers once, at creation. After the config change the old allocations kept their old addresses. It takes nomad job stop -purge then run to recreate them. That is the classic "but I already fixed it, why is it not applying".

Three settings, all holding IP addresses, all in the same file, answering three different questions: who am I, how do you reach me, under what address are my services published. None of them inherits from the others.

One more finding: MinQuorum was 0

Unrelated to the drift, found in the same audit: both autopilots ran with MinQuorum = 0 and CleanupDeadServers = true. That pair lets autopilot reap dead servers with no floor at all — three voters to two to one, each removal individually reasonable, the aggregate a single point of failure that nobody chose.

nomad operator autopilot set-config -min-quorum=3
consul operator autopilot set-config -min-quorum=3

Five minutes of work, and not free: at three servers, a floor of three also stops autopilot from cleaning up a server that is genuinely dead, so replacing one becomes a manual step. I will take that trade. It is the difference between "clean up dead servers" and "clean up dead servers, but never take the cluster below three".

The rule I wrote into the repo

In a gossip/memberlist system, the address is part of the identity. Not a property of the identity — part of it. Everything else follows:

• Never change a gossip advertise address on a live server. Not carefully, not one at a time, not with a rollback ready.
• If an entry has already drifted: stop everywhere, purge the membership cache everywhere, start everywhere. One live holder is enough to undo the whole repair, and the failure mode when it goes wrong is that the entry appears to move rather than disappear.
• Separate who I am from how you reach me in the config schema before it matters. The day the second has to change, the first must not be dragged along with it — and a single template variable feeding both guarantees that it will be.

And the real root cause is duller than any of this: two sites on the same /24. The overlay network is a workaround, an effective one, but still a workaround. Renumbering the remote site to 192.168.2.0/24 deletes the entire class of problem — the drift, the wrong-machine SSH, the fingerprinted addresses that mean two things. That is the next change, and it is three fields in a router admin page.