FIOS to FIOS connection problem
Welsberr
Enthusiast - Level 1

Sometime in the wee hours of January 10th, 2012, I lost reliable connectivity between my residential FIOS service in Florida and servers at a FIOS Business service in Texas. (The FIOS Business plan accommodates servers and provides static IP addresses.)

Things that can be transferred in transactions on the order of a single packet get through. Very short static webpages without graphics can be gotten from the server in Texas, but not much else. Any of the web applications running (Drupal, WordPress, IkonBoard, and Squirrelmail) simply timeout. This is a big problem for me because I do remote system administration on two of the servers there, and I can no longer do much of anything using SSH (even "ls" breaks the pipe used for the SSH connection). My primary email account comes through one of those servers, and I can't get to it via a client on my desktop.

But the pages and services are delivered flawlessly to everything besides FIOS that I've been able to try. I have a Verizon smartphone, and I can access email and web pages perfectly through it. The servers are serving. And FIOS to FIOS connectivity also worked without issues prior to January 10th.

The problem is replicable at other FIOS-served locations. My father usually checks in to read my weblog, but since he has Verizon FIOS, he hasn't been able to do that since January 10th, either. I've had reports of other FIOS customers in Texas not being able to get their email from the server there.

I suspect that the problem is widespread, but it is tough to verify. I've left a couple of posts on my weblog about it, but people who have FIOS aren't likely to see that until the problem gets fixed. 

So I'd like to get some more data. If you have FIOS or Verizon DSL, please try to bring up 

  http://austringer.net/wp

and let me know if you are able to see a WordPress blog hosted there, or especially if you cannot see it.

There are two tickets in on this so far. The one from the FIOS Business side is {edited for privacy}. The one from my FIOS Residential account is {edited for privacy}. Please refer to these if you enter a ticket of your own.

0 Likes
Re: FIOS to FIOS connection problem
KH-OrnEsh1
Moderator Emeritus

Please check your Private Message Inbox.; It should look like this --

0 Likes
Re: FIOS to FIOS connection problem
smith6612
Community Leader
Community Leader

Working fine here:

I'm on DSL here, BUT, here's a traceroute. Looks like I'm bumping heads with a Firewall on the Business connection's end of things.

Microsoft Windows [Version 6.0.6002]
Copyright (c) 2006 Microsoft Corporation.  All rights reserved.

C:\Users\>tracert austringer.net

Tracing route to austringer.net [71.123.242.56]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  192.168.3.1
  2    78 ms   122 ms    61 ms  10.15.3.1
  3   132 ms   103 ms   137 ms  so-1-1-0-0.BUFF-CORE-RTR1.verizon-gni.net [130.81.13.73]
  4     *       73 ms    72 ms  as4-0.NY325-BB-RTR1.verizon-gni.net [130.81.20.104]
  5   146 ms   151 ms   127 ms  P14-0-0.DLLSTX-LCR-07.verizon-gni.net [130.81.27.41]
  6     *        *        *     Request timed out.
  7     *        *        *     Request timed out.
  8     *        *        *     Request timed out.
  9  ^C
C:\Users\>

With how bad it is on your end of things, if "ls" is killing the connection, you certainly wouldn't want to accidentally type in "sl" if you happen to have that package installed (via RPM).

Anyways, perhaps the issues people have been having out in Southern California are related to what you're seeing here. Sounds just like what they've complained about. Been a huge thread going on about it on these forums, and it seems they're working on fixing it.

Re: FIOS to FIOS connection problem
Welsberr
Enthusiast - Level 1

That traceroute looks fine. This seems to be strictly a FIOS-to-FIOS connectivity problem. It's weird in several ways. One irony is that this is just about the inverse of the problem that "net neutrality" discussions bring up, in that here we have a case of a service provider who has blocked their own network from their own subscribers.

Technical support said Sunday night that they should have it resolved in 24 to 48 hours. However, I got told that back on January 10th, too. It's still broken. 

0 Likes
Re: FIOS to FIOS connection problem
Welsberr
Enthusiast - Level 1

I'm not seeing the private message inbox anywhere now. Can you be more specific?

--> Never mind, I found it.

I've posted the problem on a major science blog; hopefully I'll get some more interest there.

0 Likes
Re: FIOS to FIOS connection problem
theelkmechanic
Enthusiast - Level 1

I'm definitely seeing the same issue from here in upstate New York and have been since you first reported it. I can't figure out how to open a trouble ticket on it, though. Everything in the support site just takes me to FAQs and stuff.

Re: FIOS to FIOS connection problem
Welsberr
Enthusiast - Level 1

Unfortunately, I only know about entering tickets by calling tech support on the phone.

The blog post on Panda's Thumb helped. Several other FIOS subscribers around the country have reported the same issue. Interestingly, several others with FIOS around the country report that they are having no difficulties. A visit by tech support to the server site in Texas apparently did convince them that the problem is real and in the Verizon network, which is progress. Still no fix, but at least we're not fighting over whether it's a problem with our servers and gear now.

0 Likes
Re: FIOS to FIOS connection problem
Anthony_VZ
Master - Level 3

Sorry to hear about the problems with the connections there. If you are still having troubles connecting to it please send me a private emssage so we can look at that trouble ticket.

0 Likes
Re: FIOS to FIOS connection problem
Welsberr
Enthusiast - Level 1

Before somebody slaps a "Solved" label on this thread, I'd like to point out that the problem still continues.

Since January 10th, 2012. No change whatsoever.

1. I am the webmaster and sysadmin. I've got a MSCS degree and over twenty years experience. I do remote system administration on two servers at the Verizon FIOS Business account. Or did, until Verizon's problem beginning January 10th put an end to my making reliable connections there. Everything worked fine for nearly two years, then at about 2:10 AM EST on January 10 everything breaks. I didn't make any change then.

2. We've been through all the simple issues, tested them, and eliminated them. Ping and traceroute are not the tools needed to get at the problem.

3. We've been quite clear that the sites and email work fine for many people, including me if I use my smartphone. That sites work for some people doesn't bear any relevance to the fact that there exists a problem for other users.

4. It is not an issue at the server side with CPE. Even the Verizon techs who made the house visit have agreed on this point.

5. The fact that some Comcast subscribers may have trouble does not eliminate a Verizon problem. It's simply a more complicated problem than if Verizon dropped everyone's connection.

I've had all sorts of specious logic tossed at me on the phone and in messages, but precious little that indicates a professional customer-oriented focus on resolving issues. The tech who handled a conference call this past weekend stands out as an exception, and while I couldn't be there for it, I've had good report on the techs who actually went to the server site. The visiting Verizon techs agreed that a problem existed and that it was not in customer-provided equipment (CPE). I'm tired of others trying to tell me that it simply has to be CPE.

If the problem does eventually get fixed, I'll give a cheer for the technical folks who dug in and got after the problem. It is definitely the weirdest loss-of-service issue I've seen, and looks to be a tough problem to crack. But in the meantime, the sheer number of Verizon people looking to call it done without actually doing anything is getting depressing.

0 Likes