Skip to main content
network-tools

Traceroute Troubleshooting: How to Read the Hops That Matter

The ticket says "the site is down for some users." You open a terminal, run traceroute, and get fifteen lines of confusing output. Most developers read that output wrong — they treat every timeout as proof of an outage. Traceroute is a useful tool, but only if you know what each column actually measures and which patterns are real problems versus normal artifacts.

What traceroute actually measures

Traceroute exploits the TTL field in IP headers. Each packet starts with a TTL of 1, then 2, then 3. At every hop where the TTL hits zero, the router returns an ICMP "time exceeded" message and reveals itself. The result is a list of routers between you and the destination.

Two details people miss:

  • Each row is a reply time, not a forwarding time. The number on hop 3 is how long it took that router's control plane to answer an ICMP packet to it. Routers rate-limit and deprioritize these replies, so hop numbers are noisy samples.
  • Three probes per hop by default. Linux sends them in parallel after v7; older versions sent them one at a time. Either way, small differences between the three probes are normal. Only consistent, reproducible gaps mean something.

The four patterns that matter

  • Stars from one hop onward, never resumes. If hops 1-5 reply and hops 6+ show * * * all the way past the destination, packets stop moving at hop 5. That's a genuine break — routing drop, firewall, or a dark link.
  • Stars on a single hop, later hops reply fine. The path works; that one router just never answers ICMP probe packets. Many routers are configured this way. This pattern is cosmetic and should not trigger an escalation.
  • Latency spike that persists to the end. If hop 4 jumps from 10 ms to 180 ms and every subsequent hop stays around 180 ms, the congestion or slow link is at hop 4. The added delay carries forward.
  • Latency spike that normalizes. Hop 4 shows 200 ms but hop 5 is back to 12 ms. Almost always a rate limit or queuing artifact on that router's reply path. The actual forwarded traffic is fine.

Where traceroute misleads

Intermediate hop latency is the biggest trap on the internet. People file tickets claiming a "30 ms penalty at hop 2" when the full round trip to the server is unchanged. A reply from a control-plane CPU says nothing about forwarding performance.

Reverse paths matter too. Traceroute from A to B shows the path A-to-B for outbound traffic plus whatever path ICMP replies take back — which can be completely different. If you only trace from one direction, you see half the picture. When a provider says "our path to you is clean," ask for a traceroute from their side.

Also watch for hostnames. Router names often encode geography and interface names, like ams-ix-r1.example.net — Amsterdam, interface r1. But reverse DNS records are frequently stale or missing entirely, and a hop showing an IP from a nearby country is not proof of anything until you confirm the PTR record. Geolocation of the hop IP can also be wrong; traceroute rows are evidence, not verdicts.

A working diagnostic order

  1. Baseline with ping. Run a quick ping to the destination first — it gives you a clean end-to-end number to compare against. The free Ping Tool does this without installing anything.
  2. Trace in both directions. From your machine and, if possible, from the server back to you. Asymmetric paths are common between continents.
  3. Decode the hopnames and IPs. Paste the full output into the Traceroute Analyzer — it flags timeout patterns, summarizes latency per hop, and highlights where the path actually degrades instead of where it just looks bad.
  4. Test the port, not just the path. Traceroute confirms routers exist along the way. It says nothing about whether your service port is reachable. A firewall can forward ICMP happily and still block :5432. Check TCP reachability with the TCP Port Checker on the exact port your app uses.
  5. Rule out DNS before blaming network. Half of perceived "network outages" turn out to be DNS failures: the trace stalls at hop 1 and everything after looks like noise. Use the DNS Lookup tool to confirm the hostname still resolves to the right IP.

MPLS, tunnels, and the missing hops

Some paths simply never show every router. MPLS tunnels hide the transit path — you'll see a jump from one provider edge to another with no intermediate hops. VPNs and virtual appliances (cloud load balancers, for example) often swallow probes too. Cloud providers are the classic case: tracing to a cloud VM usually ends at the load balancer or NAT gateway, not the instance itself. If your service is behind one, traceroute can only ever prove path up to the edge — use port checks for everything beyond it.

The honest summary

Traceroute answers one question: where does the path stop behaving like a path? It does not answer whether your service is reachable, whether DNS is broken, or why packets are slow at one hop. Read the patterns, not the individual rows — resume-after-stars means ignore, spike-that-carries means investigate, and a clean trace ending past your server means move on to port and DNS checks before you blame routers.

And keep your traces. Save the output before and after any routing change on the provider side. The diff between two traces at different times is the only thing that will convince a NOC engineer that something actually changed.