×

Switch Account

TRACE ROUTE Odd Results FiOS Network

SOLVED
Reply
Highlighted
Bronze Contributor II
Bronze Contributor II
Posts: 92
Registered: ‎12-28-2011

Re: TRACE ROUTE Odd Results FiOS Network

Message 11 of 18
(2,072 Views)

Smith, alwayscurious and others - Same here, in NYC in zip code 10023, and started about the same time.

Smith - can you give us step-by-step for getting the full tracert info?  Where exactly do we put what flag in a traceroute query?

And what do the super-fast two-line results mean?  Is it possible that I got to the distant end point in only 2ms??

Highlighted
Copper Contributor
Copper Contributor
Posts: 6
Registered: ‎08-15-2012

Re: TRACE ROUTE Odd Results FiOS Network

Message 12 of 18
(2,053 Views)

Short answer: UDP and TCP works as expected, ICMP is odd.

 

MTR, all OS:  select the UDP option - mtr -u 8.8.8.8 , for example. _ mtr -buz 8.8.8.8 _ is my goto.

 

Windows: The windows tracert has no options to switch from icmp. Download a version of MTR,  install cygwin or the windows 10 bash shell environment, and use the unix traceroute, or some other solution. Not your dad.

 

Linux, Mac : See MTR above, but to observe the issue that others see and not feel left out, _ traceroute -P icmp 8.8.8.8 _

 

TMI -

It looks like they are rewriting the TTL on the outbound ICMP and the TTL on the icmp response.  Actually it looks like a TTL 1 packet hitting their network is responded by a spoof.

 

traceroute -P icmp na1.salesforce.com
traceroute: Warning: na1.salesforce.com has multiple addresses; using 136.147.103.71
traceroute to viv.l2.salesforce.com (136.147.103.71), 64 hops max, 72 byte packets
1 openwrt (192.168.1.1) 1.991 ms 1.097 ms 1.020 ms
2 dcl7-phx.viv-phx.salesforce.com (136.147.103.71) 4.577 ms 3.887 ms 5.135 ms

 

Baltimore to Phoenix and back in  ms is faster than lightspeed...

 

traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 72 byte packets
1 openwrt (192.168.1.1) 2.125 ms 1.042 ms 1.101 ms
2 google-public-dns-a.google.com (8.8.8.8) 5.915 ms 4.238 ms 4.865 ms

 

This server is an anycast server, so it is actually more like 7 MS, but still.... These packets never reach the end hosts.

 

 

ping na1.salesforce.com
PING viv.l2.salesforce.com (136.147.100.71): 56 data bytes
64 bytes from 136.147.100.71: icmp_seq=0 ttl=246 time=69.084 ms
64 bytes from 136.147.100.71: icmp_seq=1 ttl=246 time=71.081 ms
64 bytes from 136.147.100.71: icmp_seq=2 ttl=246 time=71.865 ms
64 bytes from 136.147.100.71: icmp_seq=3 ttl=246 time=70.611 ms

 

You see my point. 

70 ms makes more sense with lightspeed and all.

 

But UDP and tcp aren't munged, and pinging normally isn't messed with, so I assume there is some kind of screweyness with low TTL ICMP.

 

sudo tcpdump -nnnntttt -vvv -s2000 icmp or host 8.8.8.8
tcpdump: data link type PKTAP
tcpdump: listening on pktap, link-type PKTAP (Apple DLT_PKTAP), capture size 2000 bytes
2019-01-05 17:42:23.311669 IP (tos 0x0, ttl 1, id 37968, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 1, length 52
2019-01-05 17:42:23.312541 IP (tos 0xc0, ttl 64, id 9080, offset 0, flags [none], proto ICMP (1), length 100)
192.168.1.1 > 192.168.1.232: ICMP time exceeded in-transit, length 80
IP (tos 0x0, ttl 1, id 37968, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 1, length 52
2019-01-05 17:42:23.313559 IP (tos 0x0, ttl 1, id 37969, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 2, length 52
2019-01-05 17:42:23.314394 IP (tos 0xc0, ttl 64, id 9081, offset 0, flags [none], proto ICMP (1), length 100)
192.168.1.1 > 192.168.1.232: ICMP time exceeded in-transit, length 80
IP (tos 0x0, ttl 1, id 37969, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 2, length 52
2019-01-05 17:42:23.314476 IP (tos 0x0, ttl 1, id 37970, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 3, length 52
2019-01-05 17:42:23.315248 IP (tos 0xc0, ttl 64, id 9082, offset 0, flags [none], proto ICMP (1), length 100)
192.168.1.1 > 192.168.1.232: ICMP time exceeded in-transit, length 80
IP (tos 0x0, ttl 1, id 37970, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 3, length 52
2019-01-05 17:42:23.315298 IP (tos 0x0, ttl 2, id 37971, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 4, length 52
2019-01-05 17:42:23.320313 IP (tos 0xc0, ttl 254, id 0, offset 0, flags [none], proto ICMP (1), length 72)
8.8.8.8 > 192.168.1.232: ICMP echo reply, id 37967, seq 4, length 52
2019-01-05 17:42:23.320958 IP (tos 0x0, ttl 2, id 37972, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 5, length 52
2019-01-05 17:42:23.325093 IP (tos 0xc0, ttl 254, id 0, offset 0, flags [none], proto ICMP (1), length 72)
8.8.8.8 > 192.168.1.232: ICMP echo reply, id 37967, seq 5, length 52
2019-01-05 17:42:23.325165 IP (tos 0x0, ttl 2, id 37973, offset 0, flags [none], proto ICMP (1), length 72)
192.168.1.232 > 8.8.8.8: ICMP echo request, id 37967, seq 6, length 52
2019-01-05 17:42:23.330340 IP (tos 0xc0, ttl 254, id 0, offset 0, flags [none], proto ICMP (1), length 72)
8.8.8.8 > 192.168.1.232: ICMP echo reply, id 37967, seq 6, length 52

 

Highlighted
Contributor
Contributor
Posts: 2
Registered: ‎07-25-2017

Re: TRACE ROUTE Odd Results FiOS Network

Message 13 of 18
(1,149 Views)

That's an explanation, not a solution.  Should not be marked as such.

Still not working as of Oct 29 2019 (over a year now).

 

Highlighted
Platinum Contributor III Platinum Contributor III
Platinum Contributor III
Posts: 7,864
Registered: ‎11-04-2008

Re: TRACE ROUTE Odd Results FiOS Network

Message 14 of 18
(1,133 Views)

The solution told you how to get what you wanted.

Verizon made a change to their network.

You have options to get the same data you had before.

Highly unlikely they will change their network back.


If a forum member gives an answer you like, give them the Kudos they deserve. If a member gives you the answer to your question, mark the answer as Accepted Solution so others can see the solution to the problem.
Highlighted
Contributor
Contributor
Posts: 2
Registered: ‎07-25-2017

Re: TRACE ROUTE Odd Results FiOS Network

Message 15 of 18
(1,075 Views)

The solution told you how to get what you wanted.  -  IS IT HIDDEN?  WHERE?

 

Verizon made a change to their network. - OBVIOUSLY

 

You have options to get the same data you had before.  NO I DON'T.  ENLIGHTEN ME

 

Highly unlikely they will change their network back.  YOU KNOW THAT HOW?

Highlighted
Platinum Contributor III Platinum Contributor III
Platinum Contributor III
Posts: 5,965
Registered: ‎09-24-2008

Re: TRACE ROUTE Odd Results FiOS Network

Message 16 of 18
(1,048 Views)

@zielkj wrote:

The solution told you how to get what you wanted.  -  IS IT HIDDEN?  WHERE?

 

You have options to get the same data you had before.  NO I DON'T.  ENLIGHTEN ME

 


REF the post that is marked as solved, which is post 5 of this thread.

 

Unix/Linux OSes use, by default, UDP in their tracerouting, which seems to be fine. Windows uses ICMP by default, which is broken.

If you are the original poster (OP) and your issue is solved, please remember to click the "Solution?" button so that others can more easily find it. If anyone has been helpful to you, please show your appreciation by clicking the "Kudos" button.


 

Highlighted
Contributor
Contributor
Posts: 1
Registered: ‎06-14-2020

Re: TRACE ROUTE Odd Results FiOS Network

Message 17 of 18
(214 Views)

This is most certainly not solved, and appears to be an explicit implementation by Verizon to obscure distributed latency measurements in their network by intentionally interfering with a basic internet protocol commonly used for this purpose (ICMP).

 

Verizon spoofs (replies with a fake source address) at their first hop. They do this to ICMP packet where TTL=1.  The spoofed address incorrectly signals to your traceroute client that it has reached its destination in only a couple of hops (based on most residential network setups) at an impossibly low latency.

 

This is easy to see and work around if you have access to a linux machine with a more sophisticated traceroute that allows specification of initial TTL using the -f flag set to 3.  This results in Verizon's first hop seeing TTL=2 (one decremented by your home router).  The traceroute then proceeds but lack the first two hops (your home router, Verizon's first hop router).

 

ICMP traceroute with initial TTL=3:

traceroute -I -f 3 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
3 b3365.nycmny-lcr-22.verizon-gni.net (130.81.30.150) 5.895 ms 5.884 ms 5.885 ms
4 * * *
5 0.et-10-1-2.gw15.nyc1.alter.net (140.222.227.145) 4.103 ms 4.110 ms 4.109 ms
6 72.14.214.38 (72.14.214.38) 4.488 ms 4.491 ms 4.522 ms
7 108.170.225.6 (108.170.225.6) 4.826 ms 4.827 ms 4.874 ms
8 172.253.70.5 (172.253.70.5) 4.697 ms 2.678 ms 2.669 ms
9 dns.google (8.8.8.8) 2.195 ms 2.090 ms 2.083 ms

 

Traceroute with an initial TTL=2 showing that Verizon instantly spoofs Google's address:

 

traceroute -I -f 2 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
2 dns.google (8.8.8.8) 3.670 ms 3.704 ms 3.705 ms

 

What a normal Windows user sees and many here are reporting:

traceroute -I -f 1 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
1 gw (192.168.1.1) 0.247 ms 0.279 ms 0.280 ms
2 dns.google (8.8.8.8) 2.120 ms 2.162 ms 2.212 ms

 

This is not solved, and UDP traceroute is not a valid replacement.  Please open support tickets to Verizon about this and follow up with a complaint to the FCC here: https://consumercomplaints.fcc.gov/hc/en-us

This will not get solved without voices.

 

 

 

 

Highlighted
Gold Contributor VI Gold Contributor VI
Gold Contributor VI
Posts: 1,621
Registered: ‎12-02-2012

Re: TRACE ROUTE Odd Results FiOS Network

Message 18 of 18
(198 Views)
How-To Videos
 
The following videos were produced by users like you!
   
Videos are subject to the Verizon Fios Community Terms of Service and User Guidelines and contains content that is not created by Verizon.
Covid19


Browse Categories
Categories:
Posts

Verizon Troubleshooters
Unable to find your answer here? Try searching Verizon Troubleshooters for more options.