Re: TRACE ROUTE Odd Results FiOS Network
glnzglnz
Contributor - Level 3

Smith, alwayscurious and others - Same here, in NYC in zip code 10023, and started about the same time.

Smith - can you give us step-by-step for getting the full tracert info?  Where exactly do we put what flag in a traceroute query?

And what do the super-fast two-line results mean?  Is it possible that I got to the distant end point in only 2ms??

0 Likes
Re: TRACE ROUTE Odd Results FiOS Network
lanboyo
Newbie

Short answer: UDP and TCP works as expected, ICMP is odd.

MTR, all OS:  select the UDP option - mtr -u 8.8.8.8 , for example. _ mtr -buz 8.8.8.8 _ is my goto.

Windows: The windows tracert has no options to switch from icmp. Download a version of MTR,  install cygwin or the windows 10 bash shell environment, and use the unix traceroute, or some other solution. Not your dad.

Linux, Mac : See MTR above, but to observe the issue that others see and not feel left out, _ traceroute -P icmp 8.8.8.8 _

TMI -

It looks like they are rewriting the TTL on the outbound ICMP and the TTL on the icmp response.  Actually it looks like a TTL 1 packet hitting their network is responded by a spoof.

traceroute -P icmp na1.salesforce.com
traceroute: Warning: na1.salesforce.com has multiple addresses; using 136.147.103.71
traceroute to viv.l2.salesforce.com (136.147.103.71), 64 hops max, 72 byte packets
1 openwrt (192.168.1.1) 1.991 ms 1.097 ms 1.020 ms
2 dcl7-phx.viv-phx.salesforce.com (136.147.103.71) 4.577 ms 3.887 ms 5.135 ms

Baltimore to Phoenix and back in  ms is faster than lightspeed...

traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 72 byte packets
1 openwrt (192.168.1.1) 2.125 ms 1.042 ms 1.101 ms
2 google-public-dns-a.google.com (8.8.8.8) 5.915 ms 4.238 ms 4.865 ms

This server is an anycast server, so it is actually more like 7 MS, but still.... These packets never reach the end hosts.

ping na1.salesforce.com
PING viv.l2.salesforce.com (136.147.100.71): 56 data bytes
64 bytes from 136.147.100.71: icmp_seq=0 ttl=246 time=69.084 ms
64 bytes from 136.147.100.71: icmp_seq=1 ttl=246 time=71.081 ms
64 bytes from 136.147.100.71: icmp_seq=2 ttl=246 time=71.865 ms
64 bytes from 136.147.100.71: icmp_seq=3 ttl=246 time=70.611 ms

You see my point. 

70 ms makes more sense with lightspeed and all.

But UDP and tcp aren't munged, and pinging normally isn't messed with, so I assume there is some kind of screweyness with low TTL ICMP.

sudo tcpdump -nnnntttt -vvv -s2000 icmp or host 8.8.8.8
tcpdump: data link type PKTAP
tcpdump: listening on pktap, link-type PKTAP (Apple DLT_PKTAP), capture size 2000 bytes
2019-01-05 17:42:23.311669 IP (tos 0x0, ttl 1, id 37968, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 1, length 52
2019-01-05 17:42:23.312541 IP (tos 0xc0, ttl 64, id 9080, offset 0, flags [none], proto ICMP (1), length 100)
192.168.1.1 > 192.168.1.232: ICMP time exceeded in-transit, length 80
IP (tos 0x0, ttl 1, id 37968, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 1, length 52
2019-01-05 17:42:23.313559 IP (tos 0x0, ttl 1, id 37969, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 2, length 52
2019-01-05 17:42:23.314394 IP (tos 0xc0, ttl 64, id 9081, offset 0, flags [none], proto ICMP (1), length 100)
192.168.1.1 > 192.168.1.232: ICMP time exceeded in-transit, length 80
IP (tos 0x0, ttl 1, id 37969, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 2, length 52
2019-01-05 17:42:23.314476 IP (tos 0x0, ttl 1, id 37970, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 3, length 52
2019-01-05 17:42:23.315248 IP (tos 0xc0, ttl 64, id 9082, offset 0, flags [none], proto ICMP (1), length 100)
192.168.1.1 > 192.168.1.232: ICMP time exceeded in-transit, length 80
IP (tos 0x0, ttl 1, id 37970, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 3, length 52
2019-01-05 17:42:23.315298 IP (tos 0x0, ttl 2, id 37971, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 4, length 52
2019-01-05 17:42:23.320313 IP (tos 0xc0, ttl 254, id 0, offset 0, flags [none], proto ICMP (1), length 72)
8.8.8.8 > 192.168.1.232: ICMP echo reply, id 37967, seq 4, length 52
2019-01-05 17:42:23.320958 IP (tos 0x0, ttl 2, id 37972, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 5, length 52
2019-01-05 17:42:23.325093 IP (tos 0xc0, ttl 254, id 0, offset 0, flags [none], proto ICMP (1), length 72)
8.8.8.8 > 192.168.1.232: ICMP echo reply, id 37967, seq 5, length 52
2019-01-05 17:42:23.325165 IP (tos 0x0, ttl 2, id 37973, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 6, length 52
2019-01-05 17:42:23.330340 IP (tos 0xc0, ttl 254, id 0, offset 0, flags [none], proto ICMP (1), length 72)
8.8.8.8 > 192.168.1.232: ICMP echo reply, id 37967, seq 6, length 52

0 Likes
Re: TRACE ROUTE Odd Results FiOS Network
zielkj1
Newbie

That's an explanation, not a solution.  Should not be marked as such.

Still not working as of Oct 29 2019 (over a year now).

0 Likes
Re: TRACE ROUTE Odd Results FiOS Network
CRobGauth
Community Leader
Community Leader

The solution told you how to get what you wanted.

Verizon made a change to their network.

You have options to get the same data you had before.

Highly unlikely they will change their network back.

Re: TRACE ROUTE Odd Results FiOS Network
zielkj1
Newbie

The solution told you how to get what you wanted.  -  IS IT HIDDEN?  WHERE?

Verizon made a change to their network. - OBVIOUSLY

You have options to get the same data you had before.  NO I DON'T.  ENLIGHTEN ME

Highly unlikely they will change their network back.  YOU KNOW THAT HOW?

0 Likes
Re: TRACE ROUTE Odd Results FiOS Network
dslr595148
Community Leader
Community Leader

@zielkj wrote:

The solution told you how to get what you wanted.  -  IS IT HIDDEN?  WHERE?

You have options to get the same data you had before.  NO I DON'T.  ENLIGHTEN ME


REF the post that is marked as solved, which is post 5 of this thread.

Unix/Linux OSes use, by default, UDP in their tracerouting, which seems to be fine. Windows uses ICMP by default, which is broken.

Re: TRACE ROUTE Odd Results FiOS Network
ee16
Newbie

This is most certainly not solved, and appears to be an explicit implementation by Verizon to obscure distributed latency measurements in their network by intentionally interfering with a basic internet protocol commonly used for this purpose (ICMP).

Verizon spoofs (replies with a fake source address) at their first hop. They do this to ICMP packet where TTL=1.  The spoofed address incorrectly signals to your traceroute client that it has reached its destination in only a couple of hops (based on most residential network setups) at an impossibly low latency.

This is easy to see and work around if you have access to a linux machine with a more sophisticated traceroute that allows specification of initial TTL using the -f flag set to 3.  This results in Verizon's first hop seeing TTL=2 (one decremented by your home router).  The traceroute then proceeds but lack the first two hops (your home router, Verizon's first hop router).

ICMP traceroute with initial TTL=3:

traceroute -I -f 3 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
3 b3365.nycmny-lcr-22.verizon-gni.net (130.81.30.150) 5.895 ms 5.884 ms 5.885 ms
4 * * *
5 0.et-10-1-2.gw15.nyc1.alter.net (140.222.227.145) 4.103 ms 4.110 ms 4.109 ms
6 72.14.214.38 (72.14.214.38) 4.488 ms 4.491 ms 4.522 ms
7 108.170.225.6 (108.170.225.6) 4.826 ms 4.827 ms 4.874 ms
8 172.253.70.5 (172.253.70.5) 4.697 ms 2.678 ms 2.669 ms
9 dns.google (8.8.8.8) 2.195 ms 2.090 ms 2.083 ms

Traceroute with an initial TTL=2 showing that Verizon instantly spoofs Google's address:

traceroute -I -f 2 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
2 dns.google (8.8.8.8) 3.670 ms 3.704 ms 3.705 ms

What a normal Windows user sees and many here are reporting:

traceroute -I -f 1 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
1 gw (192.168.1.1) 0.247 ms 0.279 ms 0.280 ms
2 dns.google (8.8.8.8) 2.120 ms 2.162 ms 2.212 ms

This is not solved, and UDP traceroute is not a valid replacement.  Please open support tickets to Verizon about this and follow up with a complaint to the FCC here: https://consumercomplaints.fcc.gov/hc/en-us

This will not get solved without voices.

0 Likes
Re: TRACE ROUTE Odd Results FiOS Network
gs0b
Community Leader
Community Leader