Last week Christopher and I were the instructors of an IPv6 workshop. In this one we usually build a lab with the participants incl. a variety of routed segments and native IPv6 Internet access. Once the latter part is implemented people start poking around and surfing the Internet from their laptops, not least to find out which sites they can actually reach from an v6-only network (please note that actually there are many).
One guy had an interesting problem reaching Google’s public IPv6 DNS server (IPv6 address 2001:4860:4860::8888) though. [btw, I wonder how many ICMPv6 echo requests that address gets every day ;-)]
Here’s what he saw (while at the same time everybody else could happily ping that system):
Pinging 2001:4860:4860::8888 with 32 bytes of data:
Request timed out.
Destination net unreachable.
Request timed out.
Request timed out.
Ping statistics for 2001:4860:4860::8888:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss)
We started investigating the issue with the group (I mean, from a learning curve perspective, there’s nothing like practical troubleshooting…) and observed the following in Wireshark:
This left us wondering: why on earth does that router block innocuous ICMP Echo Requests from this specific system (the participant’s laptop)?
Here’s a traceroute output from roughly the same origin segment (I’ve highlighted the box dropping the packets):
[erey@box1 ~]$ traceroute6 -l 2001:4860:4860::8888
traceroute6 to 2001:4860:4860::8888 (2001:4860:4860::8888) from 2003:60:4010:10a0::11, 64 hops max, 12 byte packets
1 2003:60:4010:10a0::1 (2003:60:4010:10a0::1) 1.289 ms 1.206 ms 0.912 ms
2 2003:60:4010::8 (2003:60:4010::8) 0.882 ms 0.838 ms 0.772 ms
3 2003:60:4010::1 (2003:60:4010::1) 1.145 ms 1.046 ms 1.105 ms
4 2003:0:4702:c402::1 (2003:0:4702:c402::1) 6.049 ms 5.326 ms 6.443 ms
5 2003:0:130b::1 (2003:0:130b::1) 3.362 ms 3.353 ms 3.310 ms
6 2001:4860:1:1:0:cf8:0:68 (2001:4860:1:1:0:cf8:0:68) 7.480 ms 6.840 ms 6.735 ms
7 2001:4860::1:0:abf0 (2001:4860::1:0:abf0) 4.173 ms
2001:4860::1:0:abf5 (2001:4860::1:0:abf5) 4.145 ms
2001:4860::1:0:abef (2001:4860::1:0:abef) 4.413 ms
8 2001:4860::8:0:5038 (2001:4860::8:0:5038) 11.272 ms 3.762 ms
2001:4860::8:0:5039 (2001:4860::8:0:5039) 15.476 ms
9 2001:4860::8:0:8f8e (2001:4860::8:0:8f8e) 7.089 ms
2001:4860::8:0:8f91 (2001:4860::8:0:8f91) 16.191 ms 15.567 ms
As can be seen there’s only two ASs involved, that is “Deutsche Telekom AG” (DTAG) and “Google Inc.” and the (culprit) system is the last router before handing over to Google, which in turn means it’s probably a high-end device in some IX. Which then raises the question: what reason could this device have to drop said packets?
By coincidence – and luck – I recalled a situation from an IPv6-focused pentest we had performed some time ago. In the course of that one we had noticed that certain devices within the DTAG network (which at the time had to be traversed to reach the assessment customer’s network) actually dropped packets with the “Hop-by-Hop Options Header” (see RFC 2460, sect. 4.1). Taking a closer at the ICMP Echo Request packets we then saw the following:
Notice the HbH header in(serted) into the packet!
Now there’s the whole debate about blocking IPv6 extensions headers and some of you might remember that we’re in the camp of “drop’em all at your network’s border (at least if you are an enterprise organization)” (see my rant in the beginning of our RIPE70 talk ;-).
Still, there’s two questions that can be raised at this point:
- the router in question is a high-end transit device (and not an enterprise border gateway) – what could be a valid reason to drop packets with HbH and is this “allowed”/recommended as of the specs?
- what (software/mechanism) makes the packets actually carry a HbH (presumably) mostly worthless in Internet transit at all? [we think the only valid use case for HbH is MLD, running on the local-link].
As for the first question: RFC 7045 Transmission and Processing of IPv6 Extension Headers states in section 2.2:
“The IPv6 Hop-by-Hop Options header SHOULD be processed by
intermediate forwarding nodes as described in [RFC2460]. However, it
is to be expected that high-performance routers will either ignore it
or assign packets containing it to a slow processing path.”
So from that angle dropping packets with HbH can be considered “legitimate behavior” and here’s a discussion on the topic displaying the rationale and the way of handling by some parties.
This brings us to the second question: why are there HbH headers in the packets at all? The participant is using Windows 8.1 and we aren’t aware that the HbH headers gets inserted into ICMPv6 Echo Requests by default on Windows 8.1. During the break we asked two colleagues of us who are also using Windows 8.1 to ping $SOME_IPV6_DESTINATION. Interestingly, one had the same behaviour as can be seen in the screenshot:
The Windows 8.1 from the other guy didn’t exhibit the same behaviour and the ICMPv6 Echo Requests were sent without any additional extension header:
It turned out that the guy in the course and one of our colleagues actually both had Kaspersky Endpoint Protection v10 running on their boxes! Doing a little bit research, we found a thread on the Microsoft forums describing the same behaviour. The only solution they came up with was the complete deinstallation of Kaspersky. From the above we hence suspect that this is the entity adding the HbH header.
You may ask: why (does this happen)? Honestly, we’ve no idea.
We’ll try to find out and keep you posted. For the moment just take this as a little example of “IPv6 pecularities in the Internet”.
Have a great week everybody
Enno & Christopher
Want to join our team of IPv6 practitioners, work in large, challenging environments (and have quite some fun doing so, learning new things every day)? Then drop an e-mail to email@example.com.