Skip to content
Log inDashboard

GeoIP and owner classification

Two lookups enrich the query logs:

  • GeoIP maps an address to a country. It gives each answer address of the destinations analytics its country.
  • Owner classification names the company behind a query or an address: Google, Apple, Meta, Amazon and Microsoft, and the CDNs Cloudflare, Akamai and Fastly. It gives each destination its owner and feeds the top_owners analytics shape.

The control plane reads one country database in the MaxMind DB format (.mmdb). Production uses DB-IP IP to Country Lite, refreshed weekly by the geoip role (below). The code reads any of three free databases that share the format:

Database Licence Attribution the licence asks for
MaxMind GeoLite2 Country GeoLite2 EULA (account and licence key) This product includes GeoLite2 data created by MaxMind
DB-IP IP to Country Lite CC BY 4.0 IP Geolocation by DB-IP, with a link to db-ip.com
IPinfo Lite CC BY-SA 4.0 IP address data powered by IPinfo

A lookup returns the record’s country, or its registered country when it has none (anycast and satellite ranges), as an upper-case ISO 3166-1 alpha-2 code. Addresses that cannot have a country are never looked up: private, CGNAT (100.64.0.0/10), loopback, link-local, unique-local, documentation and benchmarking ranges.

opdns-cp geoip run is a long-running role (one replica is enough) that keeps the published database current:

  1. Fetch. Once a week (--geoip-interval, 168 h) it downloads the month’s DB-IP file over HTTPS (https://download.db-ip.com/free/dbip-country-lite-{YYYY}-{MM}.mmdb.gz, --geoip-dbip-url). DB-IP publishes a new file at the start of each month; until it appears, the previous month’s is used. A failed run is retried after an hour (--geoip-retry-interval).
  2. Verify. The file must open as a country database and answer the probes (--geoip-probes, default 8.8.8.8=US), and it must not be older than the one already published. A file that fails any check is refused and the published one stays.
  3. Publish. It writes the database to the opdns bucket as geoip/country.mmdb (--geoip-object), keeps the one it replaces as geoip/country.previous.mmdb, and writes the sidecar geoip/country.json: checksum, size, source, build date, attribution, when it was published and when it was last confirmed current. A run that finds the same file only updates the sidecar’s check time. On a single host, --out PATH writes a local file instead of, or as well as, the bucket.
  4. Sync. Every api and ingest replica started with OPDNS_GEOIP_OBJECT=geoip/country.mmdb reads the sidecar every OPDNS_GEOIP_SYNC_INTERVAL (10 minutes), downloads the database when its checksum changed, checks it, writes it atomically to its local OPDNS_GEOIP_DB and reloads. A replica that starts before anything is published runs without countries until the first sync.

The same binary has one-shot subcommands, which take the same flags:

Terminal window
opdns-cp geoip status # the published sidecar: build date, checksum, last check, pinned
opdns-cp geoip refresh # fetch, verify and publish now
opdns-cp geoip refresh --source file --source-file ./dbip-country-lite.mmdb.gz
opdns-cp geoip rollback # make the previous database current again, pinned
opdns-cp geoip refresh --force # publish over a pinned or newer database

--source file publishes a file obtained by other means (a DB-IP file downloaded by hand when DB-IP is unreachable from the control plane, or a MaxMind or IPinfo database). rollback swaps the current and previous databases and pins the result: the weekly run then leaves it alone (and fails, so the alert below eventually fires) until someone runs refresh --force.

Metric On Meaning
opdns_cp_geoip_last_success_timestamp_seconds geoip role when a refresh last published or confirmed the current database
opdns_cp_geoip_build_timestamp_seconds geoip role build time of the published database
opdns_cp_geoip_refresh_runs_total{result} geoip role runs by published, unchanged or error
opdns_geoip_build_timestamp_seconds api, ingest build time of the database the replica has loaded
opdns_geoip_loaded_timestamp_seconds api, ingest the replica’s last load
opdns_geoip_reload_errors_total api, ingest files that failed to load (the previous one stays in use)

GeoIPStale (warn) fires when no refresh has succeeded for 45 days; with a weekly schedule and monthly DB-IP files, a healthy database is never more than about 38 days old. A role that has never succeeded counts as stale. The GeoIP refresh runbook in the operator repository covers DB-IP being unreachable, wrong countries after a refresh (roll back) and replicas that do not pick up a new file. How alerts are routed is on Alerting.

Section titled “Configuration of the api, ingest and link roles”

The link role uses the database to place each self-hosted node where it links from, for its owner’s Analytics globe (the node’s location); it also takes OPDNS_TRUSTED_PROXIES, the load balancers whose X-Forwarded-For gives the node’s address.

Variable (flag) Default Meaning
OPDNS_GEOIP_DB (--geoip-db) none the local .mmdb path; empty or none for no database
OPDNS_GEOIP_OBJECT (--geoip-object) none the published key to sync into OPDNS_GEOIP_DB, usually geoip/country.mmdb; needs the role’s S3 settings
OPDNS_GEOIP_SYNC_INTERVAL (--geoip-sync-interval) 10 minutes how often the sidecar is checked
OPDNS_GEOIP_ATTRIBUTION (--geoip-attribution) empty the notice in GET /v1/meta: empty uses the published one (else the database type’s), none shows none, anything else is shown as is
  • Without OPDNS_GEOIP_OBJECT, a configured file must exist and load at start, or the role exits. With it, a missing file is fine until the first sync.
  • Either way the replica checks the file every 10 minutes and reloads it when its size or modification time changed, so moving a new file into place needs no restart. A file that fails to load keeps the previous database in use.
  • The local path must be writable when syncing; the images run as a non-root user, so use the allocation directory, an emptyDir or /tmp.

Without a database, every country is "": the API still returns the country column, and GET /v1/capabilities reports the destinations_map feature off.

DB-IP’s Lite licence (CC BY 4.0) requires credit wherever its data is shown. The refresh publishes the notice with the database, IP Geolocation by DB-IP (https://db-ip.com), licensed under CC BY 4.0., and the API returns the loaded database’s notice as geoip_attribution in GET /v1/meta: the published one, or else the one the database’s type asks for (GeoLite2, DB-IP or IPinfo above), or "" for any other type and without a database. The dashboard shows it beside the destinations globe, with its URLs as links. While DB-IP data is served the notice must not be empty, so do not set OPDNS_GEOIP_ATTRIBUTION=none in production.

The classifier’s data is part of the opdns source and is compiled into both the control plane and the self-hosted node, so both classify the same way with no configuration:

File Holds
domains.txt domain suffixes each company operates itself (youtube.com google, github.com microsoft, brand top-level domains such as google); the longest match wins. Curated by hand
asns.txt each owner’s autonomous systems
prefixes/<owner>.txt each owner’s address ranges, aggregated, with their sources and fetch date in the header

Where the ranges come from:

Owner Source
Google its published list (goog.json) minus Google Cloud’s customer ranges (cloud.json)
Amazon its published ip-ranges.json
Cloudflare its published IPv4 and IPv6 lists
Fastly its published public IP list
Apple, Meta, Microsoft, Akamai RIPEstat’s announced prefixes of their ASNs

A refresh script refetches the ranges. It is run by hand, never by tests or builds, and its diff is reviewed before it is committed. Changes to the classifier data need code-owner review. Tests pin well-known names and addresses (8.8.8.8 is Google’s), so a refresh that drops one shows up in review.

A query’s owner is its name’s owner (from domains.txt), or else the owner of its first answer address that falls in a known range. A destination’s owner is its address’s, by range only. Customers of a CDN are classified by address: a site behind Cloudflare counts as Cloudflare.

For cloud-stored logs, ingest adds three columns to each ClickHouse row, with the database and classifier data in use at that moment:

Column Holds
answer_country the country of each answer address, in the order of answer_ips ("" when unknown)
client_country the client address’s country, only when the profile logs client IP addresses; "" otherwise
owner the query’s owner, "" when none

Two hourly rollups follow from them: queries per owner (owners_hourly, which top_owners reads for whole-hour ranges) and answer addresses per country (countries_hourly). Like the other rollups, each expires with the rows it summarises, and both are erased with the account’s logs.

The values are frozen when the row is written: a new GeoIP database or a data update changes new rows only. Rows written before these columns existed read as "", and the API fills an empty country from its current database when it answers destinations. A destination’s owner is not read from the row: the API classifies each address when it answers, with the data it ships with.

Self-hosted nodes have no GeoIP database. A node classifies owners itself, at query time, from the data built into its release; its destinations rows carry an empty country, which the API fills from its own database as the answer passes through to the dashboard (Analytics from your node). A node older than these columns gets both added by the API.

  • Client countries follow the client IP switch. With Log client IP addresses off, the edge removes the address on the PoP, and ingest never looks up a country for the client, even when a record carries an address. With it on, the country is stored alongside the address, for the same retention. No API view returns client countries today.
  • Domain logging off means no answer countries and no owner. Such a record has no name and no answer addresses, so there is nothing to look up.
  • Lookups are local. The database is a file on the server and the classifier’s data is compiled in; no address is sent to a third party.
  • Retention and deletion. The new columns live in rows that expire per the profile’s retention; the two rollups expire with them, and account deletion erases both along with the other log tables.