Fleet operations
Every procedure here withdraws a server’s anycast route before touching
it (a drain, through the server’s anycast-guard) and announces it
again only once it is healthy, so clients move to the next PoP instead of
failing.
| Command | Does |
|---|---|
fleetctl pop add --pop P [--server S] [--site-env FILE] [--soak 60s] [--dry-run] |
a new server from inventory entry to announced |
fleetctl pop remove POP [--server S] [--grace 30s] [--dry-run] |
drain, empty and scale away a server or a PoP |
fleetctl fleet reboot --all|--wave W|POP... [--grace 30s] [--pause 30s] [--dry-run] |
reboot servers one at a time through drain |
fleetctl fleet update --image TAG --all|--wave W|POP... |
install a host release and reboot, one server at a time |
fleetctl agent status [POP...] |
the host agents: up since, release, last action |
fleetctl pop status, pop drain, pop undrain |
the guards’ state; withdraw or return a PoP by hand |
Add a PoP
Section titled “Add a PoP”fleetctl pop add takes a server that is in the fleet inventory
(validate it first with fleetctl inventory validate) to announced, in
11 steps: inventory, rendered configuration in Nomad variables (BIRD,
Unbound and the edge’s per-server settings, plus the site settings from
--site-env), the wave’s node pool, the Nomad client joining, the
pop-system job placed, the new guard held drained, the wave’s
resolver job scaled up by count only (the wave’s other servers keep
their allocations), health (20 good probes, both BGP sessions up, a live
dead-man lease), an identity check (id.server on the unicast address
names the new server), the announcement, and a soak during which it must
stay announced.
--dry-runprints what each step would write without changing anything.- It can be started before the server is up (it waits for the Nomad client) and re-run after any failure: every step checks the current state first. Each waiting step says what it is waiting on.
- Start it only when every other PoP is announced (
fleetctl pop status). - The server is first installed with the PoP bootstrap script (hardening, the Nomad client with mTLS, the host agent), and its upstream BGP session must be configured by the upstream network; step 8 waits for it.
Remove a server or a PoP
Section titled “Remove a server or a PoP”fleetctl pop remove POP drains the guard(s) server by server and waits
--grace for queries in flight, drains the Nomad node with its system
jobs and waits until nothing runs there, scales the wave’s resolver job
down by count only (checking the wave’s other allocations are untouched)
and deletes the server’s three Nomad variables. --server S removes one
server of a PoP. It refuses while another PoP is withdrawn (--force
overrides that). What is left is printed: remove the entry from the
inventory, stop the server’s Nomad client and nomad node purge it.
Reboot and host updates
Section titled “Reboot and host updates”fleetctl fleet reboot and fleetctl fleet update --image TAG walk the
selected servers in wave order, canary first, one server fleet-wide at
a time, never two of a PoP together. For each server:
- Guardrails: no
resolverrollout running, every other server announced, the server’s host agent answering and idle. - Drain, confirm BIRD withdrew the route, wait
--grace. - Act: the host agent reboots, or runs the host update with the tag and reboots only if it succeeded.
- Wait for the new boot (and, for an update, the new release).
- Health: the Nomad node ready with every allocation running, the guard healthy after 20 good probes (still drained), both BGP sessions up, a live lease.
- Undrain, wait for the announcement, pause
--pause, next server.
Any failure stops the whole run with a non-zero exit, and the server
being worked on stays drained: safe, and nothing further is touched.
Undrain it once healthy (fleetctl pop undrain) and re-run for the
servers not done yet. Every step is an event on fleet.events
(fleetctl events). Start with --wave canary, then --all.
The host agent. pop-agent runs on each server as the root systemd
unit opdns-pop-agent, outside Nomad, so a server whose Nomad client or
container runtime is broken can still be rebooted away. It answers only
status, reboot and update, only for requests naming its own server,
on fleet.agent.<pop>, and refuses to reboot or update unless its local
guard reports the server drained and withdrawn, so a replayed or mistaken
command cannot take an announced server down. It answers first and acts
two seconds later. An update runs /usr/local/sbin/opdns-pop-update TAG
and records the release in /etc/opdns-release.
Without NATS, drain with the guard’s local socket and reboot over SSH, one server at a time. Without the Nomad servers, do not reboot: see below.
The guardrails check the fleet before each server, but they cannot prevent a second, independent failure during the window: under heavy load, the guard’s health probe (400 ms, two failures) can withdraw other, healthy PoPs while one is rebooting.
fleet update --image updates the host release, not the edge (which
rolls through fleetctl rollout). A separate NATS token for pop-agent
is planned.
When the Nomad servers are down
Section titled “When the Nomad servers are down”The PoPs do not need the Nomad servers to serve DNS: allocations keep
running, the guards keep checking health and announcing or withdrawing
(BIRD is local), and fleetctl pop status, pop drain and pop undrain
work over NATS. What stops is everything that schedules or reads Nomad:
rollouts, pop add and pop remove, fleet reboot and update (they
refuse, unless --no-nomad for a server that is already dark), and any
change carried by Nomad variables (certificate or command token
rotations, per-server configuration).
During an outage: freeze deploys (no rollouts, PoP adds or removals,
certificate or token rotations), and do not reboot a PoP server: after
a reboot its Nomad client cannot restore its allocations until it reaches
a server, so it stays dark. After the servers return, check every node is
ready with the same allocation ids, then run a no-change rollout. The
full procedure is the Nomad outage runbook in the operator repository.