FiOS has completely turned me off to Verizon as an ISP
doraemon1
Enthusiast - Level 2

I am so done with this.

I can't even begin to illustrate how irritated I am with this connection and moreover Verizon's reponse.

I have no doubt I'm not alone, but that most users probably don't notice (or perhaps care) that their connection is actually exceedingly unstable.

A lot of my free time is spent working remotely on servers via SSH. This doesn't require a particular fast connection, but it does require that the **bleep** thing not drop packets right and left, which is exactly what this "FiOS" connection does.

Here is an example of some of the metrics I've collected over the past 8 weeks: http://cl.ly/3dBs

Using mtr (My Traceroute) we're able to run traceroute and review ping times and packet loss all in the same place. As you can see, one of Verizon's backbones has a serious, persistent issue. It varies from 20-50% packet loss. Verizon has been made aware of this. Their official response? "All we can do is send you a new router." Great....

I was sure a new router would fix nothing but I agreed thinking it'd at least be the new model that was rolled out earlier this year, thus replacing this piece of garbage ACTIONTEC. I was wrong. They sent exactly the same router. Come on. Are you kidding me?

They acknowledge on the phone that there is a consistent, high volume of complaints from my area, and their only response is to patronize me by sending me a new router which is the exact same hardware and firmware I was using before. Frankly, I am so disgusted with Verizon at this point I won't ever consider them for any service!

It seems Verizon's only course of action is to attempt to placate their customers with smooth-talking call-center employees, rather than addressing what is looking more and more like a serious infrastructural problem on Verizon's end.

Despite the fact that general Web browsing is mostly bearable (although this is not always the case, during periods of extreme exacerbation the network fails entirely) this level of service is unacceptable and not what I paid for. Some packet loss is to be expected, but not to the point that SSH is rendered unusable or persistent connections are inoperable.

I've been Googling around, and found that some people have had similar issues, but with no resolution. Maybe I just haven't turned over the proper leaf yet. Does anyone have tips or advice? For the immediate future I'm stuck with this connection. If I could switch to another ISP, I can tell you, I wouldn't think twice. But for now at least I can't and I'd like to see if I can at least attempt to mitigate.

0 Likes
Re: FiOS has completely turned me off to Verizon as an ISP
lasagna
Community Leader
Community Leader

Hmmm ... looking at your attached diagram, I would draw entirely different conclusions.

Your latency numbers are very weird.  You drop rates are less than 1% end to end which excepting hop #4 is not bad over an extended period of time.   Hop #4 is interesting but may well be an unrelated anomaly as the hops AFTER that hop do not exhibit similar drop rates -- which if there were a constrained node in the path, everything after a certain point would exhibit the behavior and that is clear not the case here.

Most concerning however is a 7.2% drop rate on the LOCAL LAN with a latency of over a 1000ms? -- that's unfathomable -- and points to a local issue within the network or the router as a potential source of the problem -- once which may well be masking the other difficulties.   

Have you removed all other networking devices from the router and connected -- via a good LAN cable -- to a port directly on the router and run your tests?   I would be very interested in seeing those numbers.  Have you tried a different device and/or LAN card in the machine?  What are your interface statistics on the machine?   This could be a simple speed/duplex mismatch issue on the machine's interface.

Re: FiOS has completely turned me off to Verizon as an ISP
doraemon1
Enthusiast - Level 2

@lasagna wrote:

Most concerning however is a 7.2% drop rate on the LOCAL LAN with a latency of over a 1000ms? -- that's unfathomable -- and points to a local issue within the network or the router as a potential source of the problem -- once which may well be masking the other difficulties.   


You might be misreading the metrics, which is perfectly understandable if you're not familiar with mtr.

If I run mtr on 192.168.1.1 (the router) I don't see any issues with latency whatsoever.

The reason the latency appears so horrendous at the first hop (notice also that every subsequent hop is nearly as bad or worse) is because mtr is measuring the ping time roundtrip, so we're seeing the total latency time after the data has been returned from the router to my local machine. If the connection is slow this number will /always/ increase to astronomical proportions. In effect what you're seeing is how long it takes the router to pass data to me.

Pretty nasty, eh?


@lasagna wrote:

... Hop #4 is interesting but may well be an unrelated anomaly as the hops AFTER that hop do not exhibit similar drop rates -- which if there were a constrained node in the path, everything after a certain point would exhibit the behavior and that is clear not the case here.


Unless the issue is that particular backbone, no? Remember, that's just a measurement of how the data passes through and is returned via that particular hop. So even if it's constrained, subsequent hops in and of themselves will process just fine, because there isn't anything wrong with them.


@lasagna wrote:

Have you removed all other networking devices from the router and connected -- via a good LAN cable -- to a port directly on the router and run your tests?   I would be very interested in seeing those numbers.  Have you tried a different device and/or LAN card in the machine?  What are your interface statistics on the machine?   This could be a simple speed/duplex mismatch issue on the machine's interface.


I would like to conduct further testing. 

But I can say now that this is an issue with ALL devices on the LAN, wired or wireless. It is completely indiscriminate.

0 Likes
Re: FiOS has completely turned me off to Verizon as an ISP
Hubrisnxs
Legend

are we sure we are reading this properly,  

http://en.wikipedia.org/wiki/MTR_(software)#Example

The sample MTR route, shows their first hop with no latency or loss at all.  that is what you want clearly, and the description sounds like a typical trace that isn't uncommon to most users here.

"The tool is often used for network troubleshooting. By showing a list of routers traversed, and the average round-trip time as well as packet loss to each router, it allows the user to identify links between two particular routers responsible for certain fractions of the overall latency or packet loss through the network. This can help identify network over utilization problems"

That means that if it shows packet loss on hop #1 then am I reading that incorrect?  does that not show that you have loss and latency to the lan 192.168.1.1 which sits in your house?

Re: FiOS has completely turned me off to Verizon as an ISP
doraemon1
Enthusiast - Level 2

@Hubrisnxs wrote:

That means that if it shows packet loss on hop #1 then am I reading that incorrect?  does that not show that you have loss and latency to the lan 192.168.1.1 which sits in your house?


It shows "...average round-trip time as well as packet loss" at at each hop. So, case in point is the router, that would be the latency, round-trip. In other words, it is not showing what `ping 192.168.1.1` would show, the latency between the local machine and the router.

If there's latency at any point in the WAN, that will cause concomitant latency from the router to LAN. You can't expect that the connection can be resolved before the router finishes the external requests after all. 🙂

And yes, you're right, zero loss is what I would like. I have yet to see that at any point in the six months I've been using this connection. 😞 I've gone through two routers in total, I've used multiple machines, some wired, others wireless. I've switched out the Ethernet cabling. Nothing seems to help.

Verizon has been anything but helpful to this point. The longer this goes on the more I feel that there's some kind of fundamental issue with Verizon's infrastructure that they don't want to address or perhaps can't...

Re: FiOS has completely turned me off to Verizon as an ISP
lasagna
Community Leader
Community Leader

Going to stand by my initial analysis based on the reading of the page Hubrisnxs posted.   It's basically doing a summarization of traceroute data.  Which is building it's stats based on increasing TTL values in the packet header to force ICMP time-exceeded packets to be returned.  I'll go read some more, but based on how traceroute works and that being the underlying tool, that's the only way that I see to read the statistics being displayed.

In that scenario ... ANY packet loss or latency numbers being reported between the source system and the first hop router are causes to suspect a local network issue.

What if you simply monitor the router interface with this tool (that would be a single hop) -- what do the numbers say then?

What happens if you turn off the router and do the following on the linux box:

sudo arp -d 192.168.1.1

ping 192.168.1.1

0 Likes
Re: FiOS has completely turned me off to Verizon as an ISP
doraemon1
Enthusiast - Level 2

Hi again lasagna,

In my original reply to your message I believe I mentioned that running mtr from a local machine to the router showed no latency.

Pinging the router itself is fine. Indicating issues do not exist between the router and the LAN.

You're misunderstanding mtr if you're reading it as traceroute. I believe the stock traceroute binary doesn't include ping/pack loss information, although I'm not sure of this, I do know mtr is an independent binary.

As I stated before, the latency at the first hop is most likely due to WAN issues. You will /always/ see latency, end-to-end, when there is a problem at any point in the connection. I believe I covered this in my previous posts.

I know how tempting it is to point to the LAN. But all my testing indicates that simply isn't the case. Multiple routers suggest otherwise. The same issue on all machines, wired and wireless, suggest otherwise. Metrics collected over the last eight weeks suggest otherwise. At this point, I'd need to see evidence to the contrary and I just don't.

Thanks!

0 Likes
Re: FiOS has completely turned me off to Verizon as an ISP
eljefe2
Master - Level 1

I'm curious where you're located doraemon. It seems like we're in a close geogprahical area but, FWIW, I don't see anything near the problem results you get.  This is the result from a pathping (I don't speak Linux)  to the same IP address you reported to/from: 

Microsoft Windows [Version 6.1.7600]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Users\eljefe>pathping 74.125.226.145

Tracing route to 74.125.226.145 over a maximum of 30 hops

  0 PC-Win7 [192.168.0.10]
  1  192.168.0.1
  2  L100.NYCMNY-VFTTP-119.verizon-gni.net [71.241.153.1]
  3  G2-0-3-1219.NYCMNY-LCR-12.verizon-gni.net [130.81.107.230]
  4  so-6-2-0-0.NY5030-BB-RTR1.verizon-gni.net [130.81.29.14]
  5  0.so-6-0-3.XL3.NYC4.ALTER.NET [152.63.17.85]
  6  TenGigE0-6-1-0.GW8.NYC4.ALTER.NET [152.63.21.113]
  7  google-gw.customer.alter.net [152.179.72.62]
  8  216.239.43.114
  9  216.239.48.44
 10  74.125.226.145

Computing statistics for 250 seconds...
            Source to Here   This Node/Link
Hop  RTT    Lost/Sent = Pct  Lost/Sent = Pct  Address
  0                                           jeff-PC-Win7 [192.168.0.10]
                                0/ 100 =  0%   |
  1    0ms     0/ 100 =  0%     0/ 100 =  0%  192.168.0.1
                                0/ 100 =  0%   |
  2    8ms     0/ 100 =  0%     0/ 100 =  0%  L100.NYCMNY-VFTTP-119.verizon-gni.
net [71.241.153.1]
                                0/ 100 =  0%   |
  3    9ms     0/ 100 =  0%     0/ 100 =  0%  G2-0-3-1219.NYCMNY-LCR-12.verizon-
gni.net [130.81.107.230]
                                0/ 100 =  0%   |
  4   14ms     0/ 100 =  0%     0/ 100 =  0%  so-6-2-0-0.NY5030-BB-RTR1.verizon-
gni.net [130.81.29.14]
                                0/ 100 =  0%   |
  5   17ms     0/ 100 =  0%     0/ 100 =  0%  0.so-6-0-3.XL3.NYC4.ALTER.NET [152
.63.17.85]
                                0/ 100 =  0%   |
  6   13ms     0/ 100 =  0%     0/ 100 =  0%  TenGigE0-6-1-0.GW8.NYC4.ALTER.NET
[152.63.21.113]
                                0/ 100 =  0%   |
  7   30ms     1/ 100 =  1%     1/ 100 =  1%  google-gw.customer.alter.net [152.
179.72.62]
                                0/ 100 =  0%   |
  8   15ms     0/ 100 =  0%     0/ 100 =  0%  216.239.43.114
                                0/ 100 =  0%   |
  9   11ms     0/ 100 =  0%     0/ 100 =  0%  216.239.48.44
                                0/ 100 =  0%   |
 10   14ms     0/ 100 =  0%     0/ 100 =  0%  74.125.226.145

Trace complete.

As you can see...near zero packet loss. Usually a pathping like this will be literally zero.  This particular run I lost one packet.  I've had FiOS for a year and I've never seen a problem with packet loss.  Doesn't it seem unlikely that we could both be in the NY Metro area, have such disparate results, and yet your problem is a problem with the Verizon backbone?

Re: FiOS has completely turned me off to Verizon as an ISP
doraemon1
Enthusiast - Level 2

I'm located in Queens, NY.

Here's an example of ping from my local machine to the router:

--- 192.168.1.1 ping statistics ---

1000 packets transmitted, 1000 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.824/2.723/25.867/2.977 ms

Please note that mtr, while running at the same time as this test above, displayed packet loss and latency at the first hop. Most likely this is due to WAN issues mtr was detecting at that time but because the statistics gathered by mtr are relative to the routing path it moves though all hops were affected. That is, latency is affected at all points, thus causing packet loss. I really hope this clears things up: packet loss at the first hop in mtr is not necessarily an indication of a problem between the computer running mtr and the router.

If you're still reading mtr as a regular traceroute you're misunderstanding the purpose and mechanism of the tool. HTH apropos reading the original post I made.

Thanks!

0 Likes
Re: FiOS has completely turned me off to Verizon as an ISP
Hubrisnxs
Legend

do you have a doc or a sheet about MTR that states that it works in the exact opposite way that traceroute does with respect to information gathered.  

Everything I am reading about it kinda implies the opposite, so other than taking your word for it, do you have something we can all look at so we are on the same page.    

So far we have a communication gap here, and I am sure you and everyone else would like to be on the same page when talking about this. 

http://bobcares.com/blog/?p=145   this is one source that I looked at, also the wikipedia entry above, so do you have something else that we can all look at? 

0 Likes