List rollouts

Admin only

This page is for the people who run opdns. Everything here happens on the control plane’s listc role (opdns-cp listc) and its admin port; no dashboard page exists for it.

The listc role fetches the list sources, compiles and signs the list artifact, and publishes it to object storage under lists/. A new build does not go to every PoP at once: it goes through a canary first, and any version can be rolled back fleet-wide in seconds.

The path of a build

1. A build publishes v<N> (artifact, manifest, provenance) and records it in lists/head.json. lists/latest.json, the version the fleet serves, does not move yet.
2. Canary. The role announces v<N> to the canary PoPs only: edges started with -lists-canary (OPDNS_LISTS_CANARY=true) load it, everyone else ignores it. Self-hosted nodes never receive a canary. At that moment the role records the canary PoPs’ own baseline: their block and SERVFAIL rates over a window as long as the soak, ending at the announcement (opdns_edge_queries_total by PoP and status, from Prometheus). It is stored with the rollout, so a restart during the soak keeps it, and shown per PoP as opdns_cp_listc_rollout_canary_baseline_block_ratio{pop}.
3. Soak. After the soak (--canary-soak, 15 minutes by default), the role measures the canary PoPs over the soak window and applies two thresholds to the comparison that decides:
   • the block rate may differ by at most --canary-max-block-rate-delta (default 0.05, 5 percentage points; only a rise rejects: over-blocking is what the canary catches, an under-sized build is caught by the build gates);
   • the canary’s SERVFAIL rate may rise by at most --canary-max-servfail-rate-delta (default 0.01).
4. Promotion when both hold: latest.json moves to v<N>, which is announced fleet-wide, and nodes pick it up over their link. Rejection otherwise: the canary PoPs are sent back to the served version, and the build is never promoted.

Which comparison decides, in order (every window needs --canary-min-queries answered queries, default 200, to be conclusive):

1. Self (basis: self): the canary PoPs during the soak against their own baseline. This is the primary signal, because traffic mixes differ between PoPs (in the local simulation the traffic generator reaches only the canary PoP while the others answer health probes). When both windows are conclusive it decides, whatever the other PoPs show.
2. Fleet (basis: fleet): only when the own baseline is inconclusive (the canary PoPs were quiet before the build), the canary PoPs against the other PoPs over the soak window, when both have the minimum.
3. None: otherwise the comparison is inconclusive. The role logs the three counts and extends the soak by one more window, up to --canary-max-soak-extensions times (default 3). If it is still inconclusive after the last extension, the build is promoted with a warning (“promoted without a conclusive comparison: N queries on the canary, B before the build, M elsewhere … basis: none”), opdns_cp_listc_rollout_canary_inconclusive_total counts it and ListCanaryInconclusive fires (--rollout-no-traffic=promote, the default). With --rollout-no-traffic=wait the canary soaks until traffic arrives, and ListCanaryStuck fires. A quiet fleet would otherwise reject every build; the build gate, the artifact signature and size checks and the rollback remain.

Every reason ends with the basis (for example block rate 24.98% on the canary vs 11.66% before the build (delta +13.32 pp, max 5.00); 1201 queries on the canary, 1201 before the build in 900s; basis: self), the observation and the history event carry basis, and opdns_cp_listc_rollout_decisions_total has result and basis labels. A self comparison assumes a PoP’s traffic mix is steady from one soak window to the next; a large shift in the mix just before or during the soak shows as a delta. If Prometheus cannot be reached the role waits. The very first version, and every build while the canary is off (--canary=false), is promoted directly.

State is in Postgres (list_rollouts, and every decision with its reason, actor and measured deltas in list_rollout_events). The role reconciles it every --rollout-interval (default the smaller of 15 s and a quarter of the soak), so a restart mid-rollout picks up where it was.

Flag	Environment	Default
--canary	OPDNS_CANARY	false
--canary-pops	OPDNS_CANARY_POPS	none; required with the canary
--canary-soak	OPDNS_CANARY_SOAK	15m
--canary-max-block-rate-delta	OPDNS_CANARY_MAX_BLOCK_RATE_DELTA	0.05
--canary-max-servfail-rate-delta	OPDNS_CANARY_MAX_SERVFAIL_RATE_DELTA	0.01
--canary-min-queries	OPDNS_CANARY_MIN_QUERIES	200
--canary-max-soak-extensions	OPDNS_CANARY_MAX_SOAK_EXTENSIONS	3
--rollout-no-traffic	OPDNS_ROLLOUT_NO_TRAFFIC	promote
--rollout-interval	OPDNS_ROLLOUT_INTERVAL	min(15 s, soak / 4)
--prometheus-url	OPDNS_PROMETHEUS_URL	none; required with the canary
--retain	OPDNS_RETAIN	30
--lists-next-pubkeys	OPDNS_LISTS_NEXT_PUBKEYS	none

The canary also needs NATS (--nats-url): edges learn canary and promoted versions from announcements.

Roll back

A rollback serves an older version fleet-wide at once: latest.json points at it again and it is re-announced to every PoP and node. Nodes install it even though it is older than what they run (see List updates and rollbacks). By default it also pins promotions at that version, so the next build does not undo it. The rolled-back build is marked rolled_back and never promoted again; the next build that changes something starts a new canary.

opdns-lists rollback --reason "INC-42 over-blocking" 41
opdns-lists rollback --no-pin 41      # roll back without pinning

The version must still be in storage (404 otherwise) and older than the served one (409).

Pin

A pin stops automatic promotion beyond a version without rolling back. New builds still soak on the canary PoPs, where they are held with the reason held: promotions pinned at vN. Unpin to let them through.

opdns-lists pin --reason "freeze for the migration" 41
opdns-lists unpin --reason "migration done"

Inspect

opdns-lists rollout

prints the rollout state as JSON: whether the canary is on, its PoPs, soak and thresholds, the served and head versions, the soaking canary with the reason it is still waiting, the pin, the recent rollouts and events, and comparison: the soaking (or else the last judged) canary’s baseline and the basis of its verdict.

Admin endpoints

The same operations are HTTP routes on the listc role’s admin port (the role’s --admin-addr, default :9090, which also serves /metrics and the health checks):

Route	Does
GET /admin/rollout	the rollout state
POST /admin/rollback?version=N	serve older version N fleet-wide and pin at it (&pin=false: do not pin); 404 not in storage, 409 not older than the served version
POST /admin/pin?version=N	no automatic promotion beyond N until unpinned
DELETE /admin/pin	unpin
POST /admin/build	run a build cycle now; waits up to ?wait= (default 50 s, at most 55 s): 200 done, 500 failed (the body says why), 202 still running, 409 another on-demand build running, 503 the compiler not running
GET /admin/build	the last on-demand build since the instance started (404 when none)

reason= and actor= (or the X-Opdns-Actor header) go into the audit trail; the actor defaults to admin. Access is guarded as on every role admin port: with OPDNS_ADMIN_TOKEN set, requests need Authorization: Bearer <token>; without it only loopback clients are served. See Admin tools.

opdns-lists

opdns-lists rollout | rollback | pin | unpin wrap these routes (the commands that read artifacts are below):

Flag	Environment	Default
-admin-url	OPDNS_LISTS_ADMIN_URL	http://127.0.0.1:18092 (the local simulation’s cp-listc)
-token	OPDNS_LISTS_TOKEN, then OPDNS_ADMIN_TOKEN	none
-actor	OPDNS_LISTS_ACTOR	$USER
-reason		none
-no-pin		rollback only
-timeout		30s

A version may be written 41 or v41. The command prints the JSON answer and exits non-zero with the server’s error on a failure.

List history

Since 2026-09-30 the listc role records what it fetched and what it built in Postgres (migration 0017), so the history survives the object store’s retention and can be read from the API:

• list_sources: one row per source the compiler runs or ran: its configuration, licence and attribution, the last fetch attempt (time, HTTP status, error, ETag, bytes, snapshot hash, and the kept raw snapshot’s key) and the entries it gave the last fleet build. A source removed from the configuration stays, with enabled false.
• list_versions: one row per variant (fleet, public) of every build the compiler judged: published, refused by a publish gate (gate_failed) or failed to upload (publish_error), with the artifact’s size and hash, entries per list, the source snapshots, the gate’s report and, for the fleet variant, the rollout: canary result, promotion and rollback times, and the state (canary, promoted, rejected, superseded, rolled_back).

Two API routes read them (scope admin, from a session with a second factor, or an admin token):

Route	Returns
GET /v1/admin/lists/sources	every source with its last fetch
GET /v1/admin/lists/versions	the judged builds, newest first; ?variant=fleet or public; paginated

A build refused by a gate does not use up its version number: the next build reuses it, and its row replaces the refused one.

list_versions also holds the input of the freshness panel on the opdns / lists dashboard: the time from the newest change among a fleet build’s source snapshots to its promotion, soak included (opdns_cp_listc_fetch_to_promotion_seconds). A target is planned (for example, a new threat-feed entry blocking fleet-wide within 90 minutes).

The public catalogue, GET /v1/lists, reads the same table: every list carries its toggle (the profile setting it backs), its attribution and its enabled sources, each with its URL, licence, attribution, entries and last change. It never shows the fetch errors or the sources a list no longer uses.

Rule syntax

Lists hold three kinds of rule, the same syntax as profile rules:

Rule	Matches
example.com	the name and every name below it (suffix, the default)
=example.com	the name only (exact)
*.example.com	names below it, not the name itself (children-only)

One normaliser reads every source and every in-repo file: it lowercases, converts internationalised names to A-labels, strips a trailing dot, and rejects IP literals, localhost names, single-label names and any other *. A Public Suffix List entry, from its ICANN or private section (such as co.uk), is rejected as a suffix or children-only rule, which would block a whole registry, and accepted as an exact rule. An exact and a children-only rule for one name merge into a suffix rule.

The artifact is format version 2, which stores the kind of each entry (format 1 held suffix rules only). PoPs and nodes still load format 1, so a rollback to an older build works. The byte layout is in the internal/lists package documentation.

Guard list

data/lists/guard.txt lists names that must keep resolving whatever the upstream lists say: opdns’s own domains, the DNS root and TLD infrastructure and similar. It uses the rule syntax above, with a reason after # on each line. The compiler removes every block rule, in every list and variant, that would match a guarded name: a block on a parent of a guarded name is removed whole (the artifact cannot hold a hole), a block on the same name loses the part the guard covers. Each removal (list, source, line, rule, guard rule) goes into the build’s provenance file and the cp-listc log (guard list removed a block rule), and ListGuardRemoved fires. The publish gates check the finished artifact against the guard again.

Keep guard entries specific: a guard on a broad parent disables every upstream block below it.

List ids

Profiles store list ids, so an id is assigned once and never renumbered or reused. data/lists/ids.txt is the ledger: every id ever assigned, with its name and state (active, retired, or fixture for test-only lists). A list that goes away moves to retired instead of disappearing; retired ids are listed in every manifest so that PoPs and nodes treat them as disabled. A test fails when the catalogue and the ledger disagree.

Each retired id names its replacement (replaced_by in the manifest’s retired entries): a list with the same category and profile toggle. A toggle keeps working on its own, because it switches on every list that carries it; a list enabled by id is shown in the dashboard as retired until it is turned off, and its replacement is the one to enable instead. GET /v1/lists keeps retired lists as retired: true rows.

Id	List	Retired	Why	Replaced by
8	stevenblack-unified	2026-09-30	licence: merges non-commercial upstreams (MVPS hosts, someonewhocares)	2 hagezi-normal (ads and trackers, enabled by id)
21	urlhaus	2026-09-30	licence: abuse.ch terms allow not-for-profit use only	20 hagezi-tif (threat_intel, which ingests URLhaus)
22	phishing-army	2026-09-30	licence: CC BY-NC 4.0	20 hagezi-tif (threat_intel)

guard.txt, ids.txt and gate-overrides.txt are compiled into opdns-cp, so a change reaches the listc role with the next release. Changes to data/lists/ need code-owner review.

Publish gates

Before a build is uploaded, four gates compare it with the last published build of the same variant:

Gate	Fails when
count-delta	a list of at least 100 entries loses more than half its entries, or more than quadruples
size	the artifact is over 1 GiB or 20 million entries (it is memory-mapped, so all of it ends up resident)
guard	the finished artifact still blocks a guarded name
top-sites	a site in the top 10,000 of the custom Tranco ranking becomes newly blocked by a blocklist or security list (parental lists block popular sites on purpose)

The top-sites gate compares with the previous build, so the first build of a variant only records the top sites it already blocks as its baseline. Its ranking is a custom Tranco list built from the Chrome UX Report, Majestic and Cisco Umbrella only: the default Tranco list also mixes in Cloudflare Radar (CC BY-NC) and Farsight (no stated terms), which a commercial service may not use. The listc role fetches it for this gate only; it is never compiled into a list or published. A custom list is a snapshot: it is generated again every month and its id set in internal/listc (TrancoCustomListID); until an id is set the role logs top-sites gate off and builds run without this gate. The configuration and the steps are in docs/product/list-licences.md (Tranco).

A build refused by a gate is not uploaded: the served version does not change, ListBuildGateFailed fires, opdns_listc_gate_failures_total counts it by gate, and the cp-listc log line build refused by the gate and GET /admin/build list every failure. A mistake upstream is fixed upstream (the next build passes again). An expected change goes through data/lists/gate-overrides.txt, one reviewed exception per line with a reason:

top-site example.com list=20        # reason: why this site may become blocked
count list=7 until=2026-10-15       # reason: upstream restructured the list

A top-site line lets that domain become newly blocked (by the listed lists, or by any list without list=); a count line lets that list pass its count-delta threshold until the end of that day (UTC).

Preview the gates on two builds before the role runs them:

opdns-lists gate -pub lists.pub v41.bin v42.bin
opdns-lists gate -pub lists.pub -top-sites tranco.csv @41 @42
opdns-lists gate -pub lists.pub - v42.bin       # no baseline

Both artifacts need their manifest next to them. -top-sites reads a ranking CSV in Tranco’s rank,domain form (the custom list download, zipped or not; without it the top-sites gate is off), -overrides another overrides file, and -min-entries changes the 100-entry threshold. mise run lists:gate -- ... runs the same command.

Why is a name blocked?

opdns-lists reads local artifacts, URLs, and published versions: @N is version N under -base-url (default the local simulation’s object storage) of -variant (fleet or public). Signatures are verified with -pub FILE.

opdns-lists lookup -pub lists.pub -version 42 ads.example.com
opdns-lists lookup -pub lists.pub -why @42 ads.example.com
opdns-lists diff   -pub lists.pub @41 @42
opdns-lists inspect -pub lists.pub @42
opdns-lists verify  -pub lists.pub v42.bin v42.json

Command	Prints
lookup ARTIFACT NAME	each list that matches the name, with the rule and its kind; -version N is the same as @N
lookup -why	also, from the entries sidecar, the source, snapshot hash, line and raw input line behind each match
diff OLD NEW	entry counts, then per list the rules added and removed (-limit, default 50 per list and direction, 0 for all; -counts for counts only)
inspect ARTIFACT	format, version, creation time, entries, size and lists, with the manifest when it is next to the artifact
verify ARTIFACT [MANIFEST]	checks the signature, and the manifest’s detached signature when present

mise run lists:lookup -- ... runs lookup.

Signing, provenance and retention

• The artifact is Ed25519-signed. The manifest (v<N>.json), latest.json and keys.json carry detached minisign-format signatures (.sig) that stock minisign -V verifies; latest.json names the signing key’s key_id.
• keys.json lists every key consumers should trust: current, next (announced ahead of a rotation with --lists-next-pubkeys, so consumers trust it before it signs anything) and previous (for older artifacts a rollback may still serve).
• Each build writes v<N>.provenance.json: where every source snapshot came from (URL, licence, hash, size, fetch time, HTTP validators and accepted entries, or why it was left out), the conflict report (every block rule the allow hints or the guard list removed) and the gate verdict, so a build can be traced and reproduced (with the raw snapshots under lists/raw/<sha256> while --keep-raw is on).
• Each build also writes v<N>.entries.tsv.gz, one row per entry: rule, list, source, snapshot hash and the raw input line. opdns-lists lookup -why reads it. Neither sidecar is part of the artifact.
• Retention (--retain, default 30) keeps the newest N artifacts per variant, plus the served version, its predecessor, a pinned version and a soaking canary with the version it would revert to; 0 keeps all.

List reports

Users report a list’s mistakes from a query’s details in the dashboard (Report a wrong block or a miss): a false positive (the list blocks a name it should not) or a false negative (a security list misses a harmful name). Each report records the domain, the list and its catalogue version at the time, the kind and the reporter’s note; an account may send 20 a day. Creating one is audited as list_report.create.

Triage. There is no dashboard page for the queue yet; use the API (scope admin, from a session with a second factor):

# the open queue, newest first; filter by status and kind
curl -b jar 'https://api.opdns.io/v1/admin/list-reports?status=open&kind=false_positive'
# accept or reject, with a note the reporter's organisation sees
curl -b jar -X PATCH https://api.opdns.io/v1/admin/list-reports/<id> \
  -H 'Content-Type: application/json' \
  -d '{"status":"accepted","note":"Upstream fixed it; guarded meanwhile."}'

A report can be triaged again; the latest decision stands. Each decision is audited as admin.list_report_triaged with the previous and the new status. The reporter sees the status and your note under Settings → Lists.

The reporter is emailed (template list_report_resolved) whenever a triage changes a report’s status: the domain, the list, accepted or not accepted, your note, and what happens next, with a link to the profile’s settings. Write the note for the customer. No email goes to an account without an address or awaiting deletion.

Two business days. Every report carries triage_due_at, two business days (Monday to Friday, UTC) after it was sent; the reporter sees it in the API. The api role counts the queue (opdns_cp_list_reports_open, opdns_cp_list_reports_overdue), measures each triage from report to decision (opdns_cp_list_report_triage_seconds, opdns_cp_list_reports_triaged_total{within_target}), and ListReportsOverdue fires when reports wait past the target.

Export. Accepting a report changes nothing. Turn accepted reports into candidate lines with:

opdns-cp admin list-reports export --accepted [--since 2026-09-01] [--out candidates.txt]

It reads the reports resolved since --since (a UTC day or an RFC 3339 time; default the last 7 days) and writes, to --out or stdout:

• for accepted false positives, candidate lines for data/lists/guard.txt in its syntax, several reports of one domain on one line; a domain the guard already covers is written as a comment naming the guard rule;
• for accepted false negatives, candidate top-site lines for data/lists/gate-overrides.txt. They help only when a publish gate held a list’s new block back; otherwise the fix belongs upstream, in the list itself.

A summary (reports, guard candidates, already guarded, gate-override candidates, skipped) goes to stderr. The command is read-only: a reviewer copies what stands up into data/lists/, where every change needs code-owner review like any other.

Alerts

deploy/dev/prometheus/rules/lists.rules.yml (loaded by the local simulation’s Prometheus; production loads the same file):

Alert	Severity	Fires when
ListBuildFailing	page	a variant has failed to build or publish for over an hour
ListSourceStale	warn	a source has not been refreshed for two refresh intervals
ListSourceExpired	page	a source is past its max age and left out of builds
ListCanaryStuck	warn	a canary is undecided after three soak windows (not while held by a pin)
ListCanaryRejected	warn	the canary rejected a build in the last hour
ListPromotionFailed	page	promoting, reverting or rolling back failed in the last 15 minutes
ListArtifactOld	warn	the served version was built over two days ago
ListBuildGateFailed	warn	a publish gate refused a build in the last hour; the previous version stays served
ListGuardRemoved	warn	the guard list removed block rules from the last build
ListVersionNotPromoted	warn	an edge still serves another list version 10 minutes after a promotion or rollback (an edge on the soaking canary is where it should be); check its NATS leaf and ListArtifactRefused
ListArtifactRefused	page	an edge refused an announced artifact (opdns_edge_list_refused_total{reason}: size, checksum, signature, format or version) and kept its previous version; opdns-lists verify the published version before announcing again, roll back if it is bad
ListReportsOverdue	warn	list reports have waited past the two-business-day triage target for 15 minutes

The Grafana dashboard opdns / lists shows the same series.