DNS Resolution and Troubleshooting¶

"It's always DNS" is a joke because it's usually true. This article covers how a Linux host actually resolves a hostname to an address, and the tools to find out where that process broke.

How it works¶

The resolution chain¶

When an application calls getaddrinfo("example.com") (what every DNS-aware tool uses under the hood), the C library consults /etc/nsswitch.conf:

$ grep ^hosts /etc/nsswitch.conf
hosts: files dns

This means: check /etc/hosts first, then fall back to DNS. A stale or wrong entry in /etc/hosts silently shadows real DNS and is a surprisingly common cause of "but DNS is configured correctly!" Your line may carry extra entries (on Ubuntu, for example, resolve or mdns4_minimal [NOTFOUND=return]); what matters is that files comes before dns.

On most modern systems, actual DNS queries go through systemd-resolved, not directly to the nameservers configured by your network:

$ resolvectl status
Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (eth0)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.0.2.53
       DNS Servers: 192.0.2.53 203.0.113.53

"resolv.conf mode: stub" means /etc/resolv.conf actually points at 127.0.0.53 — a local stub resolver run by systemd-resolved — rather than listing your real upstream DNS servers directly. This is normal and by design, but it means editing /etc/resolv.conf directly often has no lasting effect on these systems; it gets regenerated. Use resolvectl or your network manager's DNS settings instead.

Querying DNS directly, bypassing the OS resolver¶

dig example.com                    # full, verbose output
dig +short example.com             # just the answer
dig example.com @8.8.8.8           # query a specific server directly, bypassing /etc/resolv.conf
dig -x 203.0.113.10                # reverse lookup (PTR record)
dig example.com MX                 # a different record type

dig <name> @<server> is the single most useful DNS debugging technique: it tells you definitively whether the problem is the DNS server's answer (wrong even when queried directly) or your local resolution path (works fine against @8.8.8.8 but not through the system's normal resolver).

Reading a dig answer¶

$ dig +short example.com
203.0.113.10

With the verbose form (dig example.com, no +short), look at the ANSWER SECTION:

;; ANSWER SECTION:
example.com.        300    IN    A    203.0.113.10

The 300 is the TTL in seconds — how long resolvers are allowed to cache this answer. A DNS change that "isn't taking effect" for a while after you made it is very often just caching behaving exactly as designed, up to the old record's TTL.

Ubuntu/Debian vs. RHEL-family differences¶

On RHEL-family systems without systemd-resolved, /etc/resolv.conf usually does reflect the real upstream nameservers directly, and editing it manually works — but may be overwritten by NetworkManager on the next network event unless marked otherwise.

Practical exercise¶

1. Compare dig +short example.com against your system's default resolver vs. dig +short example.com @8.8.8.8 — confirm they return the same address (or figure out why they don't).
2. Check the TTL of the answer with plain dig example.com (without +short).
3. Deliberately add a wrong entry to /etc/hosts for a real domain, confirm it overrides DNS (getent hosts <domain> reflects the hosts-file entry), then remove it.

How to verify you did it correctly¶

• Step 3: getent hosts <domain> should return your fake IP while the entry exists in /etc/hosts, and the real DNS-resolved IP again immediately after you remove it (no caching — /etc/hosts is re-read on every lookup, unlike DNS answers).

Common mistakes¶

• Flushing the wrong cache. resolvectl flush-caches only clears systemd-resolved's cache — if your application has its own DNS cache (some runtimes/browsers do), this won't touch that.
• Troubleshooting with ping's DNS resolution instead of dig. ping does a resolution and immediately tells you if it worked, but gives none of the diagnostic detail (which server answered, the TTL, the actual record type) that dig does.
• Forgetting /etc/hosts exists at all and spending time investigating "broken" DNS servers when a stale hosts-file entry is silently shadowing them.

⚠ Dangerous commands / cautions¶

• Changing /etc/nsswitch.conf's hosts: line incorrectly (e.g., removing dns entirely) can break all hostname resolution on the host system-wide, affecting package managers, monitoring agents, and anything else that resolves names — not just your application.

Related articles¶

• TCP/IP Networking and the ip Command Suite
• Nginx Reverse Proxy and 502/504 Troubleshooting

Review questions¶

1. Why might editing /etc/resolv.conf directly have no effect on a system running systemd-resolved?
2. What does dig example.com @8.8.8.8 let you determine that a plain dig example.com doesn't?
3. Why would a DNS record change seem to "not take effect" immediately even after you've confirmed the authoritative server has the new value?
4. In what order does /etc/nsswitch.conf's default hosts: files dns consult sources, and what does that mean for a stale /etc/hosts entry?