Ping ignores the timeout in some cases

[MartinKolarik]

Multiple weird things in this measurement, reproducible for the probes/target combination:

• the ping command runs for about 30 seconds even though we set a 10-second timeout
• there is always only 1 packet with around 2 ms RTT, but it takes more than 10 seconds to appear in the output
• no other lost or awaited packets reported
• 0.000 total time

PING hk-hkg-as32167.anchors.atlas.ripe.net(2a0c:9a40:9dd7:1::2) 56 data bytes
64 bytes from 2a0c:9a40:9dd7:1::2: icmp_seq=1 ttl=60 time=2.67 ms

--- hk-hkg-as32167.anchors.atlas.ripe.net ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.665/2.665/2.665/0.000 ms

[MartinKolarik]

@SukkaW I see you own one of the two probes in the location, maybe you could try running a ping with extra debugging output to help us better understand what's happening there :

ping -6 -O -c 3 -i 0.5 -w 10 -D -v hk-hkg-as32167.anchors.atlas.ripe.net

[SukkaW]

@SukkaW I see you own one of the two probes in the location, maybe you could try running a ping with extra debugging output to help us better understand what's happening there :

ping -6 -O -c 3 -i 0.5 -w 10 -D -v hk-hkg-as32167.anchors.atlas.ripe.net

Sure! Here is the output:

root@hk-skywolf:~# ping -6 -O -c 3 -i 0.5 -w 10 -D -v hk-hkg-as32167.anchors.atlas.ripe.net
ping: sock4.fd: -1 (socktype: 0), sock6.fd: 3 (socktype: SOCK_RAW), hints.ai_family: AF_INET6

ai->ai_family: AF_INET6, ai->ai_canonname: 'hk-hkg-as32167.anchors.atlas.ripe.net'
PING hk-hkg-as32167.anchors.atlas.ripe.net(2a0c:9a40:9dd7:1::2) 56 data bytes
[1737268228.134040] 64 bytes from 2a0c:9a40:9dd7:1::2: icmp_seq=1 ident=10238 ttl=60 time=1.40 ms

--- hk-hkg-as32167.anchors.atlas.ripe.net ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.399/1.399/1.399/0.000 ms

Note that the command took very long to finish:

real    0m32.414s
user    0m0.014s
sys     0m0.006s

[SukkaW]

@MartinKolarik Since I am kinda familiar with the operator of AS7720, if this weird behavior only happens with AS7720 I can ask them if they know anything.

[SukkaW]

hk-hkg-as32167.anchors.atlas.ripe.net is pointed to 2a0c:9a40:9dd7:1::2 in IPv6. If I do ping -6 -O -c 3 -i 0.5 -w 10 -D -v 2a0c:9a40:9dd7:1::2, I can get immediate response:

ping: sock4.fd: -1 (socktype: 0), sock6.fd: 3 (socktype: SOCK_RAW), hints.ai_family: AF_INET6

ai->ai_family: AF_INET6, ai->ai_canonname: '2a0c:9a40:9dd7:1::2'
PING 2a0c:9a40:9dd7:1::2(2a0c:9a40:9dd7:1::2) 56 data bytes
[1737280085.401249] 64 bytes from 2a0c:9a40:9dd7:1::2: icmp_seq=1 ident=27775 ttl=60 time=1.31 ms
[1737280085.902607] 64 bytes from 2a0c:9a40:9dd7:1::2: icmp_seq=2 ident=27775 ttl=60 time=1.54 ms
[1737280086.403322] 64 bytes from 2a0c:9a40:9dd7:1::2: icmp_seq=3 ident=27775 ttl=60 time=1.42 ms

--- 2a0c:9a40:9dd7:1::2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 1.310/1.422/1.540/0.093 ms

But if I ping the domain (ping -6 -O -c 3 -i 0.5 -w 10 -D -v hk-hkg-as32167.anchors.atlas.ripe.net), I will stuck for 30s:

ping: sock4.fd: -1 (socktype: 0), sock6.fd: 3 (socktype: SOCK_RAW), hints.ai_family: AF_INET6

ai->ai_family: AF_INET6, ai->ai_canonname: 'hk-hkg-as32167.anchors.atlas.ripe.net'

Only after 30s (maybe kind of timeout?) I get the rest of the output:

PING hk-hkg-as32167.anchors.atlas.ripe.net(2a0c:9a40:9dd7:1::2) 56 data bytes
[1737280158.393497] 64 bytes from 2a0c:9a40:9dd7:1::2: icmp_seq=1 ident=27789 ttl=60 time=1.44 ms

--- hk-hkg-as32167.anchors.atlas.ripe.net ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.440/1.440/1.440/0.000 ms

[jimaek]

Maybe its a DNS issue? Whats the query time here?
dig hk-hkg-as32167.anchors.atlas.ripe.net

[SukkaW]

dig hk-hkg-as32167.anchors.atlas.ripe.net

Thought so, and I have already tested this with both dig and nslookup. I get instant results:

$ time dig hk-hkg-as32167.anchors.atlas.ripe.net

; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> hk-hkg-as32167.anchors.atlas.ripe.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54028
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;hk-hkg-as32167.anchors.atlas.ripe.net. IN A

;; ANSWER SECTION:
hk-hkg-as32167.anchors.atlas.ripe.net. 86400 IN A 89.213.179.65

;; Query time: 196 msec
;; SERVER: 1.0.0.1#53(1.0.0.1) (UDP)
;; WHEN: Sun Jan 19 18:09:44 HKT 2025
;; MSG SIZE  rcvd: 82


real    0m0.225s
user    0m0.020s
sys     0m0.005s


$ time nslookup hk-hkg-as32167.anchors.atlas.ripe.net
Server:         1.0.0.1
Address:        1.0.0.1#53

Non-authoritative answer:
Name:   hk-hkg-as32167.anchors.atlas.ripe.net
Address: 89.213.179.65
Name:   hk-hkg-as32167.anchors.atlas.ripe.net
Address: 2a0c:9a40:9dd7:1::2


real    0m0.233s
user    0m0.017s
sys     0m0.005s

[SukkaW]

@jimaek @MartinKolarik Working with the operator of AS7720 we have found a potential cause.

In short, even when ping a domain, the ping is still trying to find the PTR record of the resolved IP.

Note that it took Cloudflare's public DNS multiple seconds just to SERVFAIL:

$ dig -x 2a0c:9a40:9dd7:1::2 @1.0.0.1

;; communications error to 1.0.0.1#53: timed out

; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> -x 2a0c:9a40:9dd7:1::2 @1.0.0.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 62029
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; EDE: 22 (No Reachable Authority): (at delegation d.9.0.4.a.9.c.0.a.2.ip6.arpa.)
; EDE: 23 (Network Error): (66.118.228.10:53 rcode=REFUSED for 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1.0.0.0.7.d.d.9.0.4.a.9.c.0.a.2.ip6.arpa PTR)
;; QUESTION SECTION:
;2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1.0.0.0.7.d.d.9.0.4.a.9.c.0.a.2.ip6.arpa. IN PTR

;; Query time: 4180 msec
;; SERVER: 1.0.0.1#53(1.0.0.1) (UDP)
;; WHEN: Sun Jan 19 19:54:15 HKT 2025
;; MSG SIZE  rcvd: 267

The Google public DNS does return SERVFAIL almost instantly.

$ dig -x 2a0c:9a40:9dd7:1::2 @8.8.4.4

; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> -x 2a0c:9a40:9dd7:1::2 @8.8.4.4
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 18013
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
; EDE: 23 (Network Error): ([66.118.228.10] rcode=REFUSED for 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1.0.0.0.7.d.d.9.0.4.a.9.c.0.a.2.ip6.arpa/ptr)
; EDE: 23 (Network Error): ([66.118.231.10] rcode=REFUSED for 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1.0.0.0.7.d.d.9.0.4.a.9.c.0.a.2.ip6.arpa/ptr)
; EDE: 23 (Network Error): ([66.118.229.10] rcode=REFUSED for 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1.0.0.0.7.d.d.9.0.4.a.9.c.0.a.2.ip6.arpa/ptr)
; EDE: 23 (Network Error): ([66.118.230.10] rcode=REFUSED for 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1.0.0.0.7.d.d.9.0.4.a.9.c.0.a.2.ip6.arpa/ptr)
; EDE: 22 (No Reachable Authority): (At delegation d.9.0.4.a.9.c.0.a.2.ip6.arpa for 2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1.0.0.0.7.d.d.9.0.4.a.9.c.0.a.2.ip6.arpa/ptr)
;; QUESTION SECTION:
;2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1.0.0.0.7.d.d.9.0.4.a.9.c.0.a.2.ip6.arpa. IN PTR

;; Query time: 12 msec
;; SERVER: 8.8.4.4#53(8.8.4.4) (UDP)
;; WHEN: Sun Jan 19 19:57:31 HKT 2025
;; MSG SIZE  rcvd: 694

[MartinKolarik]

@SukkaW thanks! So it seems the root cause is a DNS issue but also poor handling of it in the ping tool. Can you please check if there's any difference in behavior when using the latest ping package? The easiest way to get it might be running an arch container (docker run -it --rm archlinux bash) and executing the commands inside it.

[SukkaW]

@SukkaW thanks! So it seems the root cause is a DNS issue but also poor handling of it in the ping tool. Can you please check if there's any difference in behavior when using the latest ping package? The easiest way to get it might be running an arch container (docker run -it --rm archlinux bash) and executing the commands inside it.

For some reason I can't access IPv6 in archlinux/latest:

[root@74eaa9fa844c /]# ping -6 -O -c 3 -i 0.5 -w 10 -D -v hk-hkg-as32167.anchors.atlas.ripe.net
ping: sock4.fd: -1 (socktype: 0), sock6.fd: 3 (socktype: SOCK_DGRAM), hints.ai_family: AF_INET6

ai->ai_family: AF_INET, ai->ai_canonname: 'hk-hkg-as32167.anchors.atlas.ripe.net'
ai->ai_family: AF_INET6, ai->ai_canonname: ''
ping: connect: Network is unreachable

[MartinKolarik]

Sorry, in the default configuration, it only works with the host network: docker run -it --rm --network host archlinux bash

[SukkaW]

Sorry, in the default configuration, it only works with the host network: docker run -it --rm --network host archlinux bash

On Arch Linux (which probably should have the latest ping), it does seem to work properly:

# ping -6 -O -c 3 -i 0.5 -w 10 -D -v hk-hkg-as32167.anchors.atlas.ripe.net
ping: sock4.fd: -1 (socktype: 0), sock6.fd: 3 (socktype: SOCK_RAW), hints.ai_family: AF_INET6

ai->ai_family: AF_INET6, ai->ai_canonname: 'hk-hkg-as32167.anchors.atlas.ripe.net'
PING hk-hkg-as32167.anchors.atlas.ripe.net (2a0c:9a40:9dd7:1::2) 56 data bytes
[1737421325.924254] 64 bytes from 2a0c:9a40:9dd7:1::2: icmp_seq=1 ident=6 ttl=60 time=1.39 ms

--- hk-hkg-as32167.anchors.atlas.ripe.net ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.390/1.390/1.390/0.000 ms

[MartinKolarik]

Thanks for confirming. This brings us to the problem that we haven't really considered how to update to OS-level binaries so far, but that's something for us to figure out.

[SukkaW]

Thanks for confirming. This brings us to the problem that we haven't really considered how to update to OS-level binaries so far, but that's something for us to figure out.

I can confirm the OS built-in ping is quite old:

$ ping  -V
ping from iputils 20221126
libcap: yes, IDN: yes, NLS: no, error.h: yes, getrandom(): yes, __fpending(): yes

Note that I am using Debian Bookworm on the host machine while everything is up-to-date (of Debian repository)

The Arch Linux docker image's built-in ping is very new and even supports NLS:

# ping -V
ping from iputils 20240905
libcap: yes, IDN: yes, NLS: yes, error.h: yes, getrandom(): yes, __fpending(): yes

[MartinKolarik]

Yes, we use Debian for the probes as well and they are known to ship rather old versions. But even if they shipped more updates, it wouldn't help us as we don't expect people to update our containers often.