08-22-2014 02:26 AM - last edited on 08-22-2014 06:32 AM by ElizabethS
The problem is basically part of a larger more dramatic problem with Verizon's FiOS network. Did you have a server or website go away, and not come back for a while? Thanks to another network operator, I have been able to piece this all together, and it's not good for Verizon customers.
The issue lies within a larger issue, apparently back in May, Cogent / L3 pushed a new routing table to all network operators / providers which accidentally exceeded the memory size footprint acceptable to most routers, causing a router meltdown and an alert from Cisco among other things. The same issues I was and still am experiencing with timeouts and fails on my traceroutes happened to people on nearly all Internet services, however most providers repaired their routing tables immediately. Apparently, large numbers of specifically FiOS / Verizon users, who live in several large geo-specific regions are still having severe and ongoing problems relating to the issue specifically –Baltimore, NYC, NJ, DC Metro, Los Angeles, Florida, Dallas, Atlanta and North Carolina.
It seems that my ticketing the issue, helped Verizon to identify one of the affected routing issues within their own networking that was still un-repaired, that said the issue still was and is occurring on Verizon’s network. It has nothing to do with trace or latency or ftp timeout or my modem or whatever else – it’s basically that a large amount of routers on the FiOS network are experiencing routing issues, but Verizon doesn't know which routers need table updates, and which ones don't seemingly, and those routers being spread across the nation is another factor.
The result of this problem, for me was I was being forced to wait, and wait, and wait until finally the network operation times out, hopefully bumping my IP transaction over to an unaffected router. Often, that did not happen - resulting in massive packet loss and minimally high latency time but more like total disapparance of an outside world - the internet.
Now, I am debating if I want free and clear out of my FIOS contract, but I am worried about what happens if I cancel it because so far, Verizon support refuses to admit that they failed to provide me internet – I’ll end up in collections with a balance and it will be on my credit. Many people are locked in this situation, and because it’s a ridiculous situation to have a contract of service violated by the provider, we should all be allowed a legal out is my guess.
It’s part of the larger Net Neutrality thing that Verizon probably does not wish to disclose that they've got a problem publicly to it’s customers - given ongoing support of anti-net neutrality, it looks really bad to me, that you support something in interest profit wise like market-value rates for faster speed of an IP transaction, meanwhile at the same time you know you’re failing to provide your customers internet access in many markets, much less internet at a speed acceptable to consider internet access at all.
If you believe you are affected by the problem, please note the contact points I am providing: [content removed to comply with guidelines] Make certain that you identify the issue as being related to issues ongoing with ALTER.NET - I note that your issue may or may not be ALTER.NET related ultimately so ask questions for a reply if you are unsure on determining if an issue you are having is this issue BEFORE you contact these people please - but I want Verizon support to be able to find a way to document all users reporting a problem finally and this helps Verizon.
I received a partial credit for internet service for three months, I believe it should be more, but my concern is for everyone else too, it's crazy how big the brownout is compared to how many individuals are aware of it at all.
08-22-2014 05:37 AM
If you are having the ALTER.NET issue, I might add - one of the easiest ways to figure it out is to find out if the connectivity issue falls within a brokerage to either a 69.xxx.xxx.xxx IP or a 152.xxx.xxx.xxx IP - from what I can tell, these are two of the most commonly prblematic IP ranges affected by the problem.
08-23-2014 05:24 AM - edited 08-23-2014 05:25 AM
It appears that I encountered this problem late yesterday (Albany region in NY). Some sites were not reachable (like google.com, startpage.com, verizon.com) while others worked fine (bing.com, cnn.com). I could ping these sites and get an IP and using the IP brought up the site no problem => DNS problem. I tried switching the DNS from Verizon's to OpenDNS and the unofficial Goggle DNS and that made no difference. When I did traceroutes I did see traffic going across alter.net. We did call support, but they were as clueless as always (although once in a while you catch a smart one).
As mentioned in the post above, the goal is to help Verizon weed this out. Unfortunately, the contact points were removed fromn the first post, so how do I go from here?
08-23-2014 09:53 PM
I am going to send you a private message. I need to have as many people as possible report the issue they are having to the Verizon execs they dont think there is a problem.
09-17-2014 06:37 PM - edited 09-17-2014 07:35 PM
Same problem. Problem is at only ALTER.net.
All hops before it have ZERO packet loss. ALTER.net and beyond have ~35%.
internap-gw.customer.alter.net is the faulty node.
Verizon claims no responsibility for this, apparently.
EDIT: Called again, this time they said it was their router but could not reproduce the issue. Tried it on local, and I could not produce it either. Seems to happen during ~7-10PM EST.
09-18-2014 06:54 AM
Wondering if that was cause of weird can't connect to certain common sites over last few months and to the "dropping out" sporadically on numerous computers at home here both Win and Apple.? in boston area.
09-18-2014 05:32 PM
I am in the DC suburbs in Maryland, and this issue is affecting me as well, during what I suppose are peak hours. 7pm-10pm EST sounds about right. Yesterday speedtests said I was getting 6 down, 93 up (I'm paying for 75/75). Today it's 16/33. Yesterday, by 11pm my connection was back to normal. A few hours ago, I believe it was around 6pm, my connection was normal.
09-18-2014 08:06 PM
10 minutes ago, I did a speed test at Verizon's speed test site and was only getting something like 35/90, but at 11:01pm I did another speed test, and now the connection is back to normal, like a switch had gone off. 81/91.
09-18-2014 08:22 PM - edited 09-18-2014 08:23 PM
Speeds dropping here in California like clockwork at night as well. Usually 8-Midnight Pacific Time, but have seen it slightly outside those times and during the day on the weekends. I wouldn't think that Midnight Pacific time would have a lot of traffic causing slowdowns, but maybe it is a busy time around here. Currently testing at only 15 mb upload speed at 8:20 PM even though I am on 50/50 plan. Speed was fine during the day. Download speed is usually fine, but sometimes drops at night as well.
I haven't had the issue of not being able to connect to certain websites though.
09-19-2014 09:18 PM
It happened again tonight, maybe between 8:30pm EST and 12:10am EST or thereabouts.