The handwriting of a network error

Three curl failures from one small test network:

curl: (7) Failed to connect to 198.51.100.20 port 8080 after 0 ms: Could not connect to server
curl: (7) Failed to connect to 172.31.99.10 port 9999 after 0 ms: Could not connect to server
curl: (7) Failed to connect to ghost.internal port 8080 after 3080 ms: Could not connect to server

Same exit code, same wording, three different faults: a missing route on the client, a server with nothing listening on the port, and an address no machine on the network owns. Each needs a different first command, and the line points to none of them. A fourth fault, silently dropped packets, did not even get the sentence:

curl: (28) Connection timed out after 5003 milliseconds

If your curl ends this line with the kernel’s reason, such as Connection refused, it is older than 7.86.0 (October 2022), as on Ubuntu 22.04 and RHEL 9. Newer versions end it with curl’s own summary and keep the reason for the verbose output.

What tells the cases apart is the error’s handwriting: how long the failure took, and who sent it. The time is already in the line, as after N ms or as the timeout you set. The kernel’s reason, which narrows down the sender, is one flag away, in curl -v.

The setup

All output below comes from one run: two throwaway containers on one Docker bridge network, 172.31.99.0/24. The client is Debian 13 with curl 8.14.1 at 172.31.99.5 (kernel 7.0.12-linuxkit, Docker Desktop’s VM kernel, arm64); the server is python -m http.server 8080 at 172.31.99.10. The verbose runs were made without -s, and curl’s progress-meter fragments were removed from those lines. The healthy baseline:

$ curl -sS -o /dev/null -w "HTTP %{http_code} in %{time_total}s\n" http://172.31.99.10:8080/
HTTP 200 in 0.001260s

Four faults side by side

Two rows share a time, and two rows share an errno. Neither the time nor the errno separates all four on its own; together they do.

The usual rule of thumb says a refusal takes a few to tens of milliseconds. On this bridge it took under a millisecond (curl rounds down to 0 ms), since the reset only crossed a Linux bridge between two containers on one kernel. A refusal costs one round trip to whoever sent it. Over a WAN that is typically tens of milliseconds, and time alone separates it from a missing route; on a LAN it does not, and you need the errno. The round trip also checks the sender: a refusal that arrives much faster than your ping time to the target did not come from the target.

No route: your own kernel says no

On the client, ip route add unreachable 198.51.100.0/24 (a documentation range) makes the kernel refuse that network.

$ curl -sS -m 5 http://198.51.100.20:8080/
curl: (7) Failed to connect to 198.51.100.20 port 8080 after 0 ms: Could not connect to server

$ curl -sv -m 5 http://198.51.100.20:8080/ 2>&1 | grep -iE 'trying|connect'
*   Trying 198.51.100.20:8080...
* Immediate connect fail for 198.51.100.20: No route to host
* Failed to connect to 198.51.100.20 port 8080 after 0 ms: Could not connect to server
* closing connection #0

curl calls connect() on a non-blocking socket. The kernel looks up a route before building a SYN; here it found an unreachable route and returned the error from connect() itself. “Immediate connect fail” records exactly that: nothing went on the wire, nothing to wait for. A failure decided by your own routing table takes 0 ms on any network.

Next, ask the routing table the question connect() asked:

$ ip route get 198.51.100.20
RTNETLINK answers: No route to host

On a working path it prints the route the kernel would pick. The errno depends on how the route is missing: an explicit unreachable route, as here, gives “No route to host” (EHOSTUNREACH); no matching route at all, for example without a default route, gives “Network is unreachable” (ENETUNREACH). Both come back in 0 ms, and both mean the packet never left.

Refused: the other side says no

Nothing listens on port 9999 on the server.

$ curl -sS -m 5 http://172.31.99.10:9999/
curl: (7) Failed to connect to 172.31.99.10 port 9999 after 0 ms: Could not connect to server

$ curl -sv -m 5 http://172.31.99.10:9999/ 2>&1 | grep -iE 'trying|connect'
*   Trying 172.31.99.10:9999...
* connect to 172.31.99.10 port 9999 from 172.31.99.5 port 55748 failed: Connection refused
* Failed to connect to 172.31.99.10 port 9999 after 0 ms: Could not connect to server
* closing connection #0

The headline matches the previous case, down to after 0 ms. The verbose line does not: no immediate failure this time, but a connection attempt from local port 55748 that got an answer. The SYN went out, the server’s kernel had no listener on 9999 and answered with a reset, and the client’s kernel turned the reset into ECONNREFUSED.

The sender is usually the target’s kernel. An iptables REJECT rule gives the same errno with its default icmp-port-unreachable, and tcpdump tells the two apart: a reset is TCP, that answer is ICMP, unless the rule uses --reject-with tcp-reset, which sends the reset itself. The round-trip check catches a REJECT on the path.

Next, run ss -tlnp on the target. Check the port and the bound address: a healthy service bound to 127.0.0.1 gets connections to the host’s external address refused exactly like this.

Dropped: nobody says anything

On the client, iptables -A OUTPUT -d 172.31.99.10 -p tcp --dport 8080 -j DROP discards the SYNs to the working server.

$ curl -sS -m 5 http://172.31.99.10:8080/
curl: (28) Connection timed out after 5003 milliseconds

$ curl -sv -m 5 http://172.31.99.10:8080/ 2>&1 | grep -iE 'trying|connect'
*   Trying 172.31.99.10:8080...
* Connection timed out after 5007 milliseconds
* closing connection #0

A different exit code, and no errno anywhere. No answer arrived, so the kernel kept retrying the SYN until curl’s own timer fired at the -m 5 limit. A failure that lasts exactly as long as your timeout is the handwriting of the timeout, not of the network. Without -m or --connect-timeout, the kernel’s SYN retries set the wait, about two minutes (131 s with current Linux defaults), so give every probe a limit.

Silence has several authors: a DROP rule anywhere on the path, a cloud security group, a return path that loses the replies. Check the rules closest to you (iptables -S, nft list ruleset), then retry while tcpdump -ni any host 172.31.99.10 and tcp port 8080 runs. No SYN on the wire means the drop is local: tcpdump sees outgoing packets only after the OUTPUT chain, so a SYN dropped there, as by the rule above, never appears. SYNs leaving with no reply move the search to the path and the target.

Nobody at the address: ARP gets no answer

On the client, /etc/hosts maps ghost.internal to 172.31.99.77, an address in the bridge subnet that no container holds.

$ curl -sS -m 5 http://ghost.internal:8080/
curl: (7) Failed to connect to ghost.internal port 8080 after 3080 ms: Could not connect to server

$ curl -sv -m 5 http://ghost.internal:8080/ 2>&1 | grep -iE 'trying|connect'
*   Trying 172.31.99.77:8080...
* connect to 172.31.99.77 port 8080 from 172.31.99.5 port 39396 failed: No route to host
* Failed to connect to ghost.internal port 8080 after 3071 ms: Could not connect to server
* closing connection #0

Same errno as the first case, but nothing to do with routes. 172.31.99.77 is inside the client’s own /24, so a route exists: the connected route out of eth0. To send the SYN, the kernel needed the address’s MAC, broadcast ARP requests, and got no reply. With default settings Linux asks three times, a second apart (net.ipv4.neigh.<iface>.mcast_solicit and retrans_time_ms), then marks the neighbour entry FAILED and fails the waiting connection with EHOSTUNREACH, whose text is “No route to host”. Hence the three seconds.

So ip route get would only confirm that a route exists; ask the neighbour table:

$ ip neigh show 172.31.99.77
172.31.99.77 dev eth0 FAILED

FAILED means no machine on this segment answered for the address (INCOMPLETE means the kernel is still asking). The question is now mostly inventory: is the host down, did it change address, is it on another VLAN or segment?

The same text can come from further away: a router that gets no ARP reply for a dead address behind it may send back ICMP host unreachable, and a firewall rejecting with icmp-host-prohibited does so within one round trip. Same words; a different sender, and for the REJECT a different time.

Resolving proves nothing

$ getent hosts ghost.internal
172.31.99.77    ghost.internal

The name resolved, which proves the lookup works and nothing more: nobody was at the address. curl’s headline never even showed the address: it said ghost.internal port 8080; only the Trying line revealed 172.31.99.77.

So check what a name resolves to, not whether it resolves, on the failing machine and by the path the application uses. getent goes through NSS, so it sees /etc/hosts and the rest of nsswitch.conf; prefer getent ahosts, which uses getaddrinfo(3), the call curl makes. dig asks a DNS server directly and would never have seen the hosts line that created ghost.internal. Then compare the address with where the service actually runs. Stale records and leftover hosts entries resolve perfectly well.

The 30-second procedure

1. Time first. Rerun with a limit, curl -sS -m 5 -o /dev/null <url>, and read the code in parentheses and the after N ms. Code 28 with Connection timed out at exactly your limit: the handshake never completed; go to firewall rules and tcpdump. (Operation timed out … bytes received means the connection worked and the server went quiet.) The clock starts before name resolution; with a slow resolver, put the address in the URL. Code 6: the name did not resolve; that is name resolution, not the path to the service. Code 7: note whether it took zero, one round trip, or about three seconds.
2. Then who answered. Run curl -sv -m 5 <url> 2>&1 | grep -iE 'trying|connect'. Check that the Trying address is the one you expect, then read the errno. “Immediate connect fail” means your own kernel said no inside connect(); with “No route to host” or “Network is unreachable”, that is the routing table: ip route get. “Connection refused” points at a missing listener or a REJECT rule: ss -tlnp on the target. “No route to host” after about three seconds means nobody answered ARP: ip neigh show if the address is on your segment, tcpdump for ICMP if it is not.
3. Only then a hypothesis. The usual first guess is that the service is down; in two of the four cases above, the fault was a route or a firewall rule on the client.

Reading the handwriting is a skill built on real failures, not on a table. That is what FixMyProd labs are for: a broken production system you reach from your own terminal and bring back.