Verizon at Home

[#1]

As many of you may know, a router failed in the Verizon network this afternoon causing problems for some Verizon FiOS and High Speed Internet users. Here’s what happened.

The router in question is in New York City and handles traffic from the Verizon network to the larger Internet. It is one of a number of such routers in our network.

When routers have problems, they are designed to report that they are sick. Internet traffic is rerouted to adjacent routers automatically and sent around the trouble spot. In this case, that didn’t happen. The router went into a hung state and did not appear to the rest of the network as though it was having problems.

That meant that some user traffic from the northeast continued to flow to the stalled router, but couldn’t be processed. Presto – an outage for those users.  Verizon FiOS, High Speed Internet and small business customers were affected.

The problem occurred at approximately 3:15 EDT and was detected and resolved :40 minutes later. Traffic began to flow normally and customers returned to service at that point.

Currently, we’re monitoring the router and the network watching for any other problems.

[#2]

Hi

I got the following message from my web host, hostrocket.com:

"This looks to be an issue with your ISP (Verizon). They had a key router go out earlier today which caused an outage in the NorthEast (US), which is where our datacenter is located. This has been causing trouble for customers trying to connect to our network. While they say they have the issue fixed, I've talked to some customer's that still cannot connect to our network from Verizon's. You can read about it on Verizon's site at http://forums.verizon.com/t5/Verizon-at-Home/Northeast-Router-Failure/ba-p/87001"

My IP Address Is: 173.65.xxx.xxx, I still can not connect to hostrocket.com datacenter. Could please check this as soon as possible...

Regards

Harpartap Bedi

{edited for privacy}

[#3]

The ip i am facing issues to conncet is 216.120.xxx.xxx

{edited for privacy}

[#4]

Well, I have 30 years in networking designing service provider networks and I don't have a single

design that has a single point of failure.  It appears Verizon does.

In addition to this outage, in NYC, we lost  two OC/12 SONET paths provided by

Verizon Business services to our ticker plant.

Why is it you never get the full story with Verizon or Verizon Business RFO's?

This is why I don't bother sending Verizon RFP's anymore.

This is why, despite disliking the Dolan family, when I ordered FIOS, I left my

Optimum Online Boost in place.  Both FIos and Cablevision come in to a

Juniper 5GS Firewall and terminate in to a Cisco 3350 L3 Switch (with cold spare).

Two Gig SFP's down to the upper floor that has a WS-2960G-24 via MMF.

I have full redundancy in my home network.  I design full redundancy in my

service provider networks (You don't hear the exchanges going down often right?)

Why can't Verizon build in enough capacity?

I have been using my Fios 35/20 as a primary and Optimum Boost 30M/15M

as backup verses load balancing dynamically.  I think I shall switch it around and

leave Optimum Online Live and FIOS as backup.  I have had Opt Online 9 years

with zero failures post beta test. (yea, there is some luck in that I suppose). But

had FIOS less than a year.  Had 3 outages, lost an ONT (And I have your UPS on

a true UPS!), and lost that horrible modem twice. (Modem design is the main

reason you need a firewall behind Verizon.

1. It has only WEP encryption with is crackable in 8 seconds.

2. They bridge your STB network and your Internet network

which means they can see your entire network of devices from

the broadband side.

Perhaps Verizon will replace the modem when a bunch of kids with Linux based

Netbooks walk around neighborhoods cracking the WEP and shutting down

peoples services....

How pathetic.

[#5]

I still have really spotty service ... some websites like HomeDepot, Lowes, and Facebook either timeout or load after multiple refreshes.  Verizon websites come up no problem though.

[#6]

I am having the same issue...I can't get on Amazon.com, and some other sites are slow.  I just hope that it will be resolved soon.

[#7]

i can get to theose site but twitter was donw for me and it is generally slow.

[#8]

weird thing is i can ping the site but i can't load the webpage

[#9]

I can barely connect here. Pretty much every site I try to load either times out or takes forever to load. I feel like someone should re-evaluate his/her belief that the problem has been fixed.

[#10]

Problem is still ongoing!!!!

200ms ping and 2mbps up and down with a 25/15 plan is not acceptable!

[#11]

i am getting 10/2 up and down for 50/20 plan connection is not acceptable!

[#12]

Problem still occuring in Boston area 8am on 10/3.  Spent a half-hour on the phone with FIOS support before they realized this was a network outage.  How about sending this info to the frontline?

[#13]

Having issues here in Carrollton, TX too.

[#14]

This explanation appears to be incomplete.  I am located in Portland, Oregon, and I was unable to access websites in Connecticut and North Carolina as well as in New York during the time of the outage.  Email and access to west coast sites (e.g. Yahoo, Google, Facebook) were unaffected.  I read comments by customers in western Europe who were having the same problem.  Apparently all Internet traffic into and out of the Northeast was affected.

[#15]

Still broken here in Massachusetts. It took me ten minutes to get to this page and sign in. Is there any status update?

[#16]

This outage was fixed at 5:30-5:45 this evening (Oct 3) in Massachusetts. I was told when I called Verizon yesterday at 4pm that there was a general internet outage in NY and MA, and that  'Service would be resored at 12pm (Oct 3). When I called again today - Oct 3, 3:15pm - the poor support guy went through all the diagnostics before finding the outage notice buried somewhere in his scripts - it was obviously not fromt and center of his call handling script! at 3:45pm the outage notice had an ETA of 4pm for service restoration - it was finally resolved 1hour and 15 mins later.

The original blog above stating the outage was resolved within 40 mins is rubbish and should be updated to reflect actual facts - i.e - a 28-hour utage here in MA. I'll be looking for a credit!

[#17]

I never thought I'd say this and actually mean it, but I think some of you are being way too hard on Verizon.  My company was experiencing connectivity issues yesterday.  We realized that all of our connectivity problems were related and began checking the equipment on our side (which all checked out okay).  Soon after, all connections were restored, but we didn't have an explanation.  That's when I found out about the outage that had happened for the northeast.  All in all, it wasn't a horrifically long outage and I had a reason for the outage to give my supervisor.

Even though there may have been some other issues to sort out for some people, I can confirm that at least a large part of the problem was resolved within the timeframe that is suggested by Verizon. 

For those complaining that the Verizon support frontline didn't know what was going on, you should understand that communication in even a small business can be a daunting task.  You can't be sure that every single employee will receive the communication and understand it in a timely fashion.  The frontline has to walk you through certain steps (their scripts) because that's their job.  They can do that or escalate your issue.  If I were at a higher level and an issue got escalated to me that turned out to be a low priority issue, I would be furious at the frontline.  That's why they have the scripts.  That way, the more experienced techs aren't wasting their time on trivial problems.  That said, if enough people post on this blog that they are having similar issues, I'm sure Verizon will look into it.  However, if you're having a one-off issue and posting it here, I wouldn't expect too much.  That's why they have other methods for support.  This isn't the place to be saying you're having trouble connecting to a certain IP address.  Use one of the suggested avenues for support and you'll get better results, I'm sure of it.

Redundant infrastructure is nice, but since I doubt any of us posting comments here works for Verizon, I don't think we have any place to make assumptions as to what types of redundancy Verizon has in place.  Even with redundancy, there is no way to guarantee 100% availability.  That is an impossibility, I don't care who you are.  And that percentage is including regularly scheduled failover testing.  And for that one poster, you may very well be an expert in the field, but starting your comment with how many years of experience you have only works to devalue your opinion.  Typically people who start conversations with that type of information either don't have that much experience or have been doing things the wrong way for a long time.  Leaving that tidbit out next time will help to have people take your comment more seriously.

And finally, I saw several people complaining about the quality of Verizon modems/routers.  Seriously?  I don't think anyone is holding a gun to your head telling you that you must use that equipment.  If you'd like a more secure router, then go get one.  When I had FIOS, I used their device for the STB network and my own router for everything else.  Sure, WEP can be cracked quickly, but it's still more secure than probably 90% of most home wireless networks.  You know, the ones that still have the out-of-the-box configurations.  If by some chance some Linux h@x0r with mad skillz does own your network, don't expect it to influence your ISP's equipment purchasing decisions.  That would be like blaming the liquor store because you got arrested for driving under the influence of alcohol.  Stuff just happens sometimes.

Don't get me wrong, Verizon has a lot of areas in which they need to improve.  There are many people that work for them that will only serve to frustrate you, but there is a time and place to discuss all that.  Comments here should be related to the outage that occurred yesterday and not about every single thing you hate about Verizon.  My apologies to anyone who did just that...plainly stated your related issue and moved on.  That's how it should be done.