High Latency to first HOP
chhatc
Enthusiast - Level 2

Hi All,

I'm seeing big variations in my first hop ping and want to bring this to the attention of Verizon techs so that they can help me sort out the reasons for the latencies. Here's some background.

I run my own ethernet cable (cat5e shielded) from the GPON Alcatel ONT to my own router. I own a Buffalo WZR-HP-AG300H Dual band router wifi with gigabit ethernet (wan/lan are all g-bit). Currently the Buffalo runs on OpenWrt and it is rock stable and hasn't needed to be rebooted in weeks. That being said, I have also rebooted the router several times in order to see if the problem goes away to no avail.

I can see that the router connects to the ONT at 1Gbps. No matter what the router indicates my IP Address/Gateway combination are, the first hop is always the same IP (and it is not my indicated gateway). The first hop IP is always 98.113.8.1.

I have run several tests throughout the day and I've measured the delays to the first hop. While most times it is well under 5ms, it spikes as high as 1000+ ms. I can vouch for the fact there there are absolutely no other causes internet traffic/latency spikes. I've narrowed everything down to the sole connection and it see the spikes irrespective of any download/upload activity. I have another Buffalo router (same model) running stock firmware and the pings are identical.

Any idea what I causing these spikes and is there something that I can do to make sure the spikes are gone?

EDIT: These tests are from a PC running Ubuntu 12.04.2 using a wired port on the router (1gbps connection negotiated). No changes in Windows 7 mode either (except that the ping data isn't so granulated).

PING to 192.168.1.1 (router):

@Zoidberg:~$ ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_req=1 ttl=64 time=0.280 ms
64 bytes from 192.168.1.1: icmp_req=2 ttl=64 time=0.261 ms
64 bytes from 192.168.1.1: icmp_req=3 ttl=64 time=0.247 ms
64 bytes from 192.168.1.1: icmp_req=4 ttl=64 time=0.254 ms
64 bytes from 192.168.1.1: icmp_req=5 ttl=64 time=0.245 ms
64 bytes from 192.168.1.1: icmp_req=6 ttl=64 time=0.233 ms
64 bytes from 192.168.1.1: icmp_req=7 ttl=64 time=0.265 ms
64 bytes from 192.168.1.1: icmp_req=8 ttl=64 time=0.241 ms
64 bytes from 192.168.1.1: icmp_req=9 ttl=64 time=0.261 ms
64 bytes from 192.168.1.1: icmp_req=10 ttl=64 time=0.253 ms
64 bytes from 192.168.1.1: icmp_req=11 ttl=64 time=0.242 ms
64 bytes from 192.168.1.1: icmp_req=12 ttl=64 time=0.264 ms
64 bytes from 192.168.1.1: icmp_req=13 ttl=64 time=0.262 ms
64 bytes from 192.168.1.1: icmp_req=14 ttl=64 time=0.241 ms
64 bytes from 192.168.1.1: icmp_req=15 ttl=64 time=0.278 ms
64 bytes from 192.168.1.1: icmp_req=16 ttl=64 time=0.226 ms
64 bytes from 192.168.1.1: icmp_req=17 ttl=64 time=0.281 ms
64 bytes from 192.168.1.1: icmp_req=18 ttl=64 time=0.267 ms
64 bytes from 192.168.1.1: icmp_req=19 ttl=64 time=0.266 ms
64 bytes from 192.168.1.1: icmp_req=20 ttl=64 time=0.272 ms
64 bytes from 192.168.1.1: icmp_req=21 ttl=64 time=0.278 ms
64 bytes from 192.168.1.1: icmp_req=22 ttl=64 time=0.259 ms
64 bytes from 192.168.1.1: icmp_req=23 ttl=64 time=0.272 ms
64 bytes from 192.168.1.1: icmp_req=24 ttl=64 time=0.279 ms
64 bytes from 192.168.1.1: icmp_req=25 ttl=64 time=0.232 ms
64 bytes from 192.168.1.1: icmp_req=26 ttl=64 time=0.226 ms
64 bytes from 192.168.1.1: icmp_req=27 ttl=64 time=0.266 ms
64 bytes from 192.168.1.1: icmp_req=28 ttl=64 time=0.258 ms
64 bytes from 192.168.1.1: icmp_req=29 ttl=64 time=0.200 ms
^C
--- 192.168.1.1 ping statistics ---
29 packets transmitted, 29 received, 0% packet loss, time 27998ms
rtt min/avg/max/mdev = 0.200/0.255/0.281/0.024 ms

Here's how I figure my first hop:

zoidberg@Zoidberg:~$ traceroute gmail.com
traceroute to gmail.com (74.125.226.214), 30 hops max, 60 byte packets
1 192.168.1.1 (192.168.1.1) 0.169 ms 0.169 ms 0.212 ms
2 L100.NYCMNY-VFTTP-84.verizon-gni.net (98.113.8.1) 4.729 ms 4.835 ms 5.064 ms
3 G1-15-0-6.NYCMNY-LCR-22.verizon-gni.net (130.81.223.30) 9.274 ms 9.305 ms 9.326 ms
4 ae2-0.NY5030-BB-RTR2.verizon-gni.net (130.81.199.170) 10.562 ms 10.561 ms 10.572 ms
5 0.so-0-0-0.XT2.NYC4.ALTER.NET (152.63.17.21) 6.628 ms 6.642 ms 6.665 ms
6 TenGigE0-7-2-0.GW8.NYC4.ALTER.NET (152.63.21.129) 12.087 ms TenGigE0-5-2-0.GW8.NYC4.ALTER.NET (152.63.16.125) 6.502 ms TenGigE0-5-0-0.GW8.NYC4.ALTER.NET (152.63.21.65) 5.795 ms
7 * * *
8 209.85.255.68 (209.85.255.68) 6.485 ms 6.552 ms 6.521 ms
9 72.14.239.250 (72.14.239.250) 8.263 ms 8.680 ms 8.441 ms
10 lga15s28-in-f22.1e100.net (74.125.226.214) 6.384 ms 6.426 ms 6.376 ms

Ping GMAIL.COM

@Zoidberg:~$ ping gmail.com
PING gmail.com (74.125.226.214) 56(84) bytes of data.
64 bytes from lga15s28-in-f22.1e100.net (74.125.226.214): icmp_req=1 ttl=250 time=6.70 ms
64 bytes from lga15s28-in-f22.1e100.net (74.125.226.214): icmp_req=2 ttl=250 time=3.95 ms
64 bytes from lga15s28-in-f22.1e100.net (74.125.226.214): icmp_req=3 ttl=250 time=5.22 ms
64 bytes from lga15s28-in-f22.1e100.net (74.125.226.214): icmp_req=4 ttl=250 time=5.82 ms
64 bytes from lga15s28-in-f22.1e100.net (74.125.226.214): icmp_req=5 ttl=250 time=3.40 ms
64 bytes from lga15s28-in-f22.1e100.net (74.125.226.214): icmp_req=6 ttl=250 time=4.21 ms
64 bytes from lga15s28-in-f22.1e100.net (74.125.226.214): icmp_req=7 ttl=250 time=5.80 ms
64 bytes from lga15s28-in-f22.1e100.net (74.125.226.214): icmp_req=8 ttl=250 time=3.87 ms
64 bytes from lga15s28-in-f22.1e100.net (74.125.226.214): icmp_req=9 ttl=250 time=4.24 ms
64 bytes from lga15s28-in-f22.1e100.net (74.125.226.214): icmp_req=10 ttl=250 time=5.29 ms
64 bytes from lga15s28-in-f22.1e100.net (74.125.226.214): icmp_req=11 ttl=250 time=6.57 ms
64 bytes from lga15s28-in-f22.1e100.net (74.125.226.214): icmp_req=12 ttl=250 time=4.42 ms
64 bytes from lga15s28-in-f22.1e100.net (74.125.226.214): icmp_req=13 ttl=250 time=7.33 ms
64 bytes from lga15s28-in-f22.1e100.net (74.125.226.214): icmp_req=14 ttl=250 time=4.52 ms
64 bytes from lga15s28-in-f22.1e100.net (74.125.226.214): icmp_req=15 ttl=250 time=4.68 ms
64 bytes from lga15s28-in-f22.1e100.net (74.125.226.214): icmp_req=16 ttl=250 time=5.26 ms
^C
--- gmail.com ping statistics ---
16 packets transmitted, 16 received, 0% packet loss, time 15023ms
rtt min/avg/max/mdev = 3.409/5.084/7.336/1.093 ms

PING TO FIRST HOP:

@Zoidberg:~$ ping 98.113.8.1
PING 98.113.8.1 (98.113.8.1) 56(84) bytes of data.
64 bytes from 98.113.8.1: icmp_req=1 ttl=126 time=5.60 ms
64 bytes from 98.113.8.1: icmp_req=2 ttl=126 time=2.93 ms
64 bytes from 98.113.8.1: icmp_req=3 ttl=126 time=4.18 ms
64 bytes from 98.113.8.1: icmp_req=4 ttl=126 time=3.03 ms
64 bytes from 98.113.8.1: icmp_req=5 ttl=126 time=2.09 ms
64 bytes from 98.113.8.1: icmp_req=6 ttl=126 time=3.45 ms
64 bytes from 98.113.8.1: icmp_req=7 ttl=126 time=2.12 ms
64 bytes from 98.113.8.1: icmp_req=8 ttl=126 time=0.988 ms
...

64 bytes from 98.113.8.1: icmp_req=20 ttl=126 time=27.8 ms

...

64 bytes from 98.113.8.1: icmp_req=29 ttl=126 time=1.10 ms
...

64 bytes from 98.113.8.1: icmp_req=37 ttl=126 time=145 ms
...

64 bytes from 98.113.8.1: icmp_req=42 ttl=126 time=0.962 ms
...

64 bytes from 98.113.8.1: icmp_req=47 ttl=126 time=147 ms
...

64 bytes from 98.113.8.1: icmp_req=51 ttl=126 time=23.2 ms
...

64 bytes from 98.113.8.1: icmp_req=57 ttl=126 time=115 ms
...

64 bytes from 98.113.8.1: icmp_req=67 ttl=126 time=108 ms
...

64 bytes from 98.113.8.1: icmp_req=77 ttl=126 time=86.7 ms
...

64 bytes from 98.113.8.1: icmp_req=87 ttl=126 time=78.3 ms
...

64 bytes from 98.113.8.1: icmp_req=97 ttl=126 time=59.2 ms
...

64 bytes from 98.113.8.1: icmp_req=102 ttl=126 time=25.1 ms
...

64 bytes from 98.113.8.1: icmp_req=107 ttl=126 time=131 ms
...

64 bytes from 98.113.8.1: icmp_req=117 ttl=126 time=31.2 ms
...

64 bytes from 98.113.8.1: icmp_req=127 ttl=126 time=19.0 ms
...

64 bytes from 98.113.8.1: icmp_req=130 ttl=126 time=40.6 ms
...

64 bytes from 98.113.8.1: icmp_req=142 ttl=126 time=18.4 ms
...

64 bytes from 98.113.8.1: icmp_req=281 ttl=126 time=122 ms
...

64 bytes from 98.113.8.1: icmp_req=340 ttl=126 time=25.1 ms
64 bytes from 98.113.8.1: icmp_req=341 ttl=126 time=18.8 ms
64 bytes from 98.113.8.1: icmp_req=342 ttl=126 time=22.5 ms
64 bytes from 98.113.8.1: icmp_req=343 ttl=126 time=21.0 ms
64 bytes from 98.113.8.1: icmp_req=344 ttl=126 time=19.9 ms
64 bytes from 98.113.8.1: icmp_req=345 ttl=126 time=18.9 ms
64 bytes from 98.113.8.1: icmp_req=346 ttl=126 time=17.8 ms
64 bytes from 98.113.8.1: icmp_req=347 ttl=126 time=16.0 ms
64 bytes from 98.113.8.1: icmp_req=348 ttl=126 time=14.3 ms
64 bytes from 98.113.8.1: icmp_req=349 ttl=126 time=13.8 ms
64 bytes from 98.113.8.1: icmp_req=350 ttl=126 time=11.7 ms
64 bytes from 98.113.8.1: icmp_req=351 ttl=126 time=10.5 ms
64 bytes from 98.113.8.1: icmp_req=352 ttl=126 time=9.84 ms
...

64 bytes from 98.113.8.1: icmp_req=386 ttl=126 time=23.2 ms
64 bytes from 98.113.8.1: icmp_req=387 ttl=126 time=3.93 ms
64 bytes from 98.113.8.1: icmp_req=388 ttl=126 time=1.50 ms
64 bytes from 98.113.8.1: icmp_req=389 ttl=126 time=2.63 ms
64 bytes from 98.113.8.1: icmp_req=390 ttl=126 time=3.61 ms
64 bytes from 98.113.8.1: icmp_req=391 ttl=126 time=4.52 ms
64 bytes from 98.113.8.1: icmp_req=392 ttl=126 time=1.94 ms
64 bytes from 98.113.8.1: icmp_req=393 ttl=126 time=2.75 ms
64 bytes from 98.113.8.1: icmp_req=394 ttl=126 time=5.82 ms
64 bytes from 98.113.8.1: icmp_req=395 ttl=126 time=4.15 ms
64 bytes from 98.113.8.1: icmp_req=396 ttl=126 time=2.09 ms
64 bytes from 98.113.8.1: icmp_req=397 ttl=126 time=4.17 ms
64 bytes from 98.113.8.1: icmp_req=398 ttl=126 time=3.09 ms
64 bytes from 98.113.8.1: icmp_req=399 ttl=126 time=3.20 ms
64 bytes from 98.113.8.1: icmp_req=400 ttl=126 time=4.29 ms
^C
--- 98.113.8.1 ping statistics ---
400 packets transmitted, 400 received, 0% packet loss, time 399567ms
rtt min/avg/max/mdev = 0.962/6.691/147.471/17.220 ms

Any ideas?

0 Likes
Re: High Latency to first HOP
click4dylan
Contributor - Level 1
We have the same problem in Florida, for a long time actually. Every time we try to talk to Verizon about it we get the runaround. They have even sent techs out who don't even know what a trace route is. One time I managed to get to level 2 who apparently was convinced it was something on my network when clearly it wasn't. Do not bother trying to get help with it. Your results are actually better than here. I have posted trace routes on this forum showing the first hop and sometimes the second or third Verizon hop with over 300 ms consistently.
0 Likes
Re: High Latency to first HOP
tns2
Community Leader
Community Leader

HIGH LATENCY.  You must be burying it in the numbers, be cause all I see is outstanding response times and very low latency.  Note that many of these servers also setup to use low priority on responding, so 1000+ (1 second) is not unheard of without actually effecting real response time to your real world servers.

0 Likes
Re: High Latency to first HOP
chhatc
Enthusiast - Level 2

I agree the latency isn't near as high as 1000+ ms, and yes it wasn't one of the results of this ping test. It was a result on one of my previous tests which I wasn't able to salvage.

What I can agree with is that my ping responses for google.com are fairly consistent. Hence, your answer with respect to verizon nodes having lower ICMP response priority makes a whole lot of sense. I do have good pings to sites such as google.com, dslreports.com, etc as a result of converting to Ethernet.

It might simply be that verizon interconnections simply have a lower priority on ICMP echo tests.

Point taken.

Thanks,

chhatc

0 Likes