Consistent packet loss just a few hops into Verizon's network

[shuep]

I have had consistent packet loss just a few hops into Verizon's network for some time. I have tried the normal things like rebooting routers and releasing my ip address but none of that helps. Most routes do not have an issue, but there is one particular service in Verizon's network which I have issues with, and it happens to be the server that Verizon routes me through when I am gaming online.

The server is : G0-3-3-6.RCMDVA-LCR-22.verizon-gni.net (130.81.191.80) and I average around 50% packet loss each night. It is just the second hop into my route:

1) My Router Internal IP

2) L100.RCMDVA-VFTTP-16.verizon-gni.net (98.117.88.1) with a ping of 9 and 0 packet loss

3) G0-3-3-6.RCMDVA-LCR-22.verizon-gni.net (130.81.191.80)  with a ping of 15 and 50-80% packet loss.

How can I get Verizon to stop routing me through this troublesome server? If I look at the route to goggle.com for example, I don't go through that server and there is no packet loss at all.

[devnuller]

Send a full traceroute... or better yet, download pingplotter or mtr and show the end to end results

[JOSEm441]

This is more of a question than a suggested fix, Would changing the DNS server help out in this situation? There is a short article here http://piyushlabs.wordpress.com/2012/01/02/why-should-you-change-your-dns-server/ that has some basic information. I am using google DNS servers and have no loss reported with pathping command.

[smith6612]

---

@JOSEm441 wrote:

This is more of a question than a suggested fix, Would changing the DNS server help out in this situation? There is a short article here http://piyushlabs.wordpress.com/2012/01/02/why-should-you-change-your-dns-server/ that has some basic information. I am using google DNS servers and have no loss reported with pathping command.

---

Changing the DNS server may help in the case that a site is using a CDN, where you could be sent to a different server, taking a different path. Otherwise, changing DNS Servers won't help if the traffic must go through those points. DNS is simply a phone book of names and IP addresses. It's not a router or a proxy.

[shuep]

Here is WinMTR (and lower than normal packet pass on the third hop, it is often 50% and above.

WinMTR Statistics

WinMTR statistics

Host	%	Sent	Recv	Best	Avrg	Wrst	Last
192.168.1.1	0	102	102	0	0	1	0
L100.RCMDVA-VFTTP-16.verizon-gni.net	0	102	102	6	15	160	7
G0-3-3-6.RCMDVA-LCR-22.verizon-gni.net	25	52	39	9	15	24	15
130.81.163.66	0	102	102	9	20	105	12
0.xe-11-1-0.BR1.IAD8.ALTER.NET	0	102	102	10	15	59	24
ae-10.r04.asbnva02.us.bb.gin.ntt.net	0	102	102	58	64	69	66
ae-12.r00.asbnva02.us.bb.gin.ntt.net	14	66	57	63	69	79	72
ae14.bbr02.eq01.wdc02.networklayer.com	0	102	102	59	63	73	65
ae7.bbr01.eq01.wdc02.networklayer.com	0	102	102	62	66	104	65
ae0.bbr01.tl01.atl01.networklayer.com	10	74	67	0	45	55	45
ae13.bbr02.eq01.dal03.networklayer.com	14	66	57	51	61	96	63
ae1.dar02.sr01.dal01.networklayer.com	0	102	102	51	55	67	56
po2.fcr05.sr06.dal01.networklayer.com	0	102	102	50	54	131	52
173.192.44.22-static.reverse.softlayer.com	0	102	102	49	54	64	54

And then ping Plotter for a little later:

As you can see I have problems within the Verizon network, and ONLY with that one server. When I ping google or other sites, Verizon sends me through a different server and I have no problems. 

[shuep]

I am still having this problem, the packet loss from that same verizon hop is up to 94% tonight. I tried using a friends VPN service to see if I could play without lag, and it worked perfectly; as long as I am routed around the faulty server, I have no issues at all. However, I am already paying Verizon plenty of money, there is no reason I should have to pay for a VPN just to route around a server they should fix.

I take it there is no chance of getting help here? There is no email address for verizon, and I would rather not waste time with people telling me to reboot my router.

[Skwishie]

I just got FIOS two days ago and experienced similar packet loss for two straight nights. I called and did the typical restart etc stuff to no avail. My problem is a little different from yours in that I was dropping packets to my gateway, the first hop, not the second. 

What eventually fixed the problem was removing my old cisco router that I was using previously (solely as a wifi hub so I did not have to re-enter all the wifi passwords moving to a new ISP) from the Verizon FIOS router. I am unsure if there are compatibility issues via routing protocols or what have you, but my packet loss disappeared once I removed the secondary router.

It does not make much sense to me why my old router would cause packet loss when connected to the Verizon FIOS router/internet box, but if you have a similar setup, I thought I would throw out that idea. Good luck. 

[shuep]

I just have the ActionTec router that Verizon provided. I also don't have packet loss problems off peak, it is always around 8-10pm EST and it is always just that one server in Verizon's network, the second hop. I wonder if it is some kind of traffic shaping or throttling that I am getting caught up in.

[mrballcb]

I aggree with you that t's likely a severely congested link you are being routed through.  8-10 PM fits in with that generic time where family members are going to have fired up their favorite streaming video source(s).  Since it's a congested link, you also should be aware that ICMP (what ping and traceroute use) become nearly useless for determining packet loss.  A router backplane assigns a grade to every packet that wants to cross from one network connection to another, and ICMP typically is assigned low grade/value.  So when a router is congested and needing to drop packets, ICMP is one of the first ones to get dropped.  Less than 1% of your TCP traffic may be having problems, but ICMP failure might be greater than 50%.  Find a traceroute program that can use TCP to do the traceroute for more accurate results.  I'm not a Windows guy, so I have no clue what programs you have which can do this.

[dslr595148]

---

@mrballcb wrote:

Since it's a congested link, you also should be aware that ICMP (what ping and traceroute use) become nearly useless for determining packet loss.  A router backplane assigns a grade to every packet that wants to cross from one network connection to another, and ICMP typically is assigned low grade/value.  So when a router is congested and needing to drop packets, ICMP is one of the first ones to get dropped.  Less than 1% of your TCP traffic may be having problems, but ICMP failure might be greater than 50%.  Find a traceroute program that can use TCP to do the traceroute for more accurate results.  I'm not a Windows guy, so I have no clue what programs you have which can do this.

---

On my Linux/Unix computer, I use tcptraceroute

On a Windows computer, you need to download and install tracetcp.