Uploads of large files always fail on FiOS

[cu_deh]

About 2 weeks ago, I noticed that my computer was always humming. Turned out that DropBox was stuck trying to upload some files.  The files in question still haven't uploaded.

The long and the short of it is this: I have some good ideas about what's going on but have so far been unable to get Verizon to let me communicate with someone who has hopes of having of understanding what I'd like to communicate.  I probably need to talk with a L2 or L3 tech, but haven't figured out how to accomplish that, and Verizon can't seem to figure out how to fix the problem.

The problem:

• Uploads of tiny files (< 100 bytes) are likely to succeed.
• Uploads of files in the 20KB range succeed sometimes.
• As the file size approaches 1MB, the probability of success approaches 0.
• Doesn't matter how the device is connected to the router, or what site you're trying to upload to--Google Drive, DropBox, Yahoo Mail, Verizon Mail, etc.
• Our Android phones experience this when on the WiFi network, but work seamlessly when on the T-Mobile network--part of the proof that this is a Verizon network problem.
• There is a growing list of neighbors who also have FiOS Internet (some with as much as 75/35 speeds) and are experiencing the same problem.  Speed tests show that the upload (and download) speeds are close to what is expected, but uploads just don't work.
• Downloads work fine, and ping/traceroute tests look good.  WinMTR shows no problems whatsoever.
• But when I used wireshark to capture the packets while attempting to upload an attachment to mail.verizon.com, I could see what was happening.  The TCP connection got established, but the server was sending lots of duplicate ACKs.  This got out of hand as time progressed, and eventually the TCP stack on my computer gave up and shutdown the connection.  It is quite likely that one of 2 things is happening.  (1) Outbound packets are being corrupted and so being dropped at the receiver--the verizon mail server in this case--when the checksum fails in the TCP/IP stack.  Uploads of larger files use larger packet sizes and the probability of corruption is increasing with packet size.  (2) Outbound packets are being lost/dropped in the Verizon network for some reason, which again is happens with higher probability during larger uploads.

I've noticed other posts which sound similar but which do not describe how the problem was resolved:

• Uploading-Issues
• No Upload on Fios

I've probably wasted 3 to 4 hours on the phone with Verizon, as well as 5 hours at my home watching over a technician who readily admitted he hadn't a clue and failed in his effort to get connected with a network tech who was knowledgeable enough to help.  The router and ONT were changed in my house.  The PON card in the OLT was reset, etc.  Of course, none of this helped.  Who knows what other tinkering they've done.  I can only say that uploads still fail just the same and now downloads seem a bit more sluggish.

Any ideas?

[Hubrisnxs]

Take a look at this, for what will probably be a good solution for you.

All Large (~3GB+) Files Download Corrupt

Also this one.  

---

kes601 join:2007-04-14 Virginia Beach, VA

reply to drumz0rz Somebody else had similar issues and it turned out a new Actiontec was the only thing that fixed it: »[Northeast] Problems since upgrading speed

---

drop by a vz store and pick one up on the way home.
»www22.verizon.com/home/storelocator/

[cu_deh]

@Hubrisnxs: Thanks for the suggestions.  I think my situation is fundamentally differently than the ones you've mentioned.

In particular:

• upload vs. download
• problems encountered by TCP vs. by an app tried to read the downloaded file (i.e. its not that the file gets uploaded and then we notice it is corrupt--it won't upload)
• problem affects multiple homes vs a single home

That said, I think I've already covered the suggested solutions.  I actiontec router has been replaced with a more recent variant.  We tried using the repair tech's laptop directly connected to the actiontec, and tried using Ethernet rather than Coax to go from the ONT to the actiontec.

The evidence suggests that the problem is most likely in 1 of 2 places:

1. At the PON card in the OLT
2. At the 1st hop router after the OLT

[Hubrisnxs]

Do you know what model ONT you have (might be common to the area) and does a reboot of the ONT ever fix the problem?  

As far as getting in contact with someone that might understand, is difficult, and in my opinion best acheived with a tech on site.  Tech's on site have internal contacts for Tier II departments and network technician's for issues like this.    Plus those departments usually want to work with field technicians to do testing and isolation

[cu_deh]

Looks like I have the Tellabs 612A ONT. Haven't tried rebooting it myself, but the ONT was replaced during the visit (which effectively accomplished the same).

I had also concluded that the best way to get in touch is was when a tech is on site, so I had one come out today.  Unfortunately, no L2 network techs were available on the weekend.  We'll try again on Monday.  In the mean time, I'm continuing to learning about Verizon's PON architecture.

Thanks for your help and attention.

[acpl]

Hi cu_deh,

Did you fix this problem? I have the exactly same issue ever since I switched from Cox to FIOS in September and I am unable to upoload any large files (> 2mb). This is driving me crazy becasue I never had any problems with Cox.

[cu_deh]

The problem was finally fixed yesterday, by replacing the PON card.

For the record, here are the symptoms that were most interesting to me:

• Downloads worked fine.  Uploads of files > 1 MB or so always failed regardless of the target site (Google, Dropbox, Yahoo, ...)

• Speed tests to nearest servers generally reported achieving the provisioned download speed, but only 60-70% of the provisioned upload speed.
• Speed tests to slightly more distant servers often failed with: Upload test returned an error while trying to read the upload file.

• Traceroute and ping (or winmtr) did not reveal any major issues.

• Packet traces collected of upload attempts using Wireshark (Capture > Interfaces ... Start; then enter "tcp" in the Filter Expression box and hit Apply) showed successful TCP connection establishment followed by lots of duplicate Acks.  (Don't even think about this if you're not technically inclined.)

• When I setup an IPSec VPN tunnel (which sends encrypted packets over UDP/IP to a peer at my employer's site) and added routes so that the target of my upload would be routed over the tunnel, things worked if not perfectly.  I was definitely able to upload files. (Ignore this one as well if it doesn't immediately make sense or you don't already have an IPSec VPN to play with.  If you do, this may be a makeshift workaround for accomplishing a few limited tasks.)

What I found most helpful in getting a resolution:

1. Avoid the problem-is-with-the-target-site deflection (e.g. Yahoo or DropBox) as early as possible.  I did this by making my test case against Verizon's own mail server (login to your account, Emails > Compose, Attach ... <some large file>).
2. Get a technician dispatched to your home as soon as possible, and let them go through the paces (plug a PC in directly to the router, reset/replace the router, reset/replace the ONT, etc.).
3. If possible, find another neighbor who has Verizon FiOS and is also experiencing the same problems.  Get the phone/account # associated with their service (with their permission) and pass that info along.
4. Once you've established that the problem is outside of your house and applies to other neighbors, it is most likely either with the FDH or the PON card.  At this point you probably need the L2 network  techs to act.  Calling the 800 number is pretty useless in trying to get their attention, unless you call the 800 number to get a technician dispatched to your house so that the L2 techs can have a body on site to work with.   And its worthy noting that they go home at 6pm (Eastern time?) and don't work weekends.  What you need is this phone number: 1-855-682-8623.  May not apply to the entire country, but was appropriate for the northeast.

Here's hoping this helps at least one other person get a little faster resolution that we experienced.

[skillset1]

I had this exact same problem for about 10 days: Small files would upload ok, but anything over 2mb would fail after some time.  Had multiple techs out - both very good - and both indicated a CO (central office) issue.  My problem was also accompanied by clicking on the phone that only callers could hear (I could not).  

Another tech was supposed to come out tomorrow to take some more stats before escalating to L2 - sure wish this was easier to escalate than it has been.  But, service was interrupted tonight for a few hours (mostly the phone was dead - but had some internet and tv issues) which made me finally believe the problem was getting solved.  It finally is, I'm very happy to report.  I attached this thread to my ticket initially hoping the PON card replacement idea could help them.

Glad this issue is fixed as i was contemplating switching back to cable if my upload and phone issues couldn't be fixed.  Verizon really needs to be able to escalate issues much sooner and they should NEVER close a ticket without verifying an issue has been resolved.  They closed multiple tickets on me and they shouldn't not have.  Good news is that the phone techs and techs that were onsite treated the issue like it should have been - a Verizon issue versus PC or house issue which I was quite certain I did not have.

I love Fios but sometimes Verizon can be tough.