Extremely Reduced Download (Upload, Ping both OK)

[mattgoff]

For the past week (maybe two), I've had severely reduced download speeds.  I'm on a 15/5 plan and speedtest.net shows that I'm getting between 500kbps and 1Mpbs down, but ~5Mbps up (normal) and pings in the 35-45ms range to a server ~100 mi away (pingtest.net shows grade A too).  The download rate chart shows that b/w varies a lot during the download.

This is confirmed with perfmon (unfortunately limited to 1 sec resolution).  In general, the test starts off strong with a few ticks @ near full bandwidth, then drops to practically nothing.  Perfmon sample resolution is too short to get the full b/w, so every d/l sample is showing lower than u/l in the below.

Performance tests done on verizon.net/speedtest show normal (which isn't helping me convice L1 to escalate me).  Since pings are fine in the above tests, tracert doesn't show anything interesting.

Here's where it gets interesting.  I ran tests with WinMTR to a bunch of different major sites: google.com, microsoft.com, akamai.com, my own personal webhost, and they all show moderate (~10%) packet loss on hosts outside the VZ network.

Here's an example set from akamai.com:

WinMTR statistics

Host	%	Sent	Recv	Best	Avrg	Wrst	Last
myrouter.home	0	107	107	0	0	0	0
L100.TAMPFL-VFTTP-88.verizon-gni.net	0	107	107	6	8	42	10
G0-9-4-7.TAMPFL-LCR-21.verizon-gni.net	0	107	107	10	15	66	12
130.81.199.26	0	107	107	10	17	123	11
0.xe-5-0-0.BR3.ATL4.ALTER.NET	6	85	80	26	33	103	30
204.255.168.222	12	72	64	26	30	61	29
atx-edge-03.inet.qwest.net	0	107	107	27	35	115	29
208.46.17.170	12	71	63	28	30	47	29

One more from google.com which actually does show some loss within the VZ network:

WinMTR statistics

Host	%	Sent	Recv	Best	Avrg	Wrst	Last
myrouter.home	0	101	101	0	0	1	0
L100.TAMPFL-VFTTP-88.verizon-gni.net	0	101	101	5	7	12	6
G0-9-4-7.TAMPFL-LCR-22.verizon-gni.net	7	80	75	11	13	31	11
so-2-0-0-0.TPA01-BB-RTR2.verizon-gni.net	5	83	79	11	18	148	22
0.ge-3-2-0.XL4.MIA4.ALTER.NET	7	78	73	15	22	116	17
0.so-6-0-2.XL4.IAD8.ALTER.NET	7	81	76	48	53	117	50
TenGigE0-7-2-0.GW7.IAD8.ALTER.NET	5	83	79	42	50	89	44
google-gw.customer.alter.net	11	69	62	91	107	126	115
216.239.46.248	4	87	84	42	52	140	92
72.14.238.173	13	65	57	50	53	65	55
74.125.228.7	9	74	68	45	47	55	50

I keep going around and around with L1 support who won't escalate the issue.  I've been playing along, repeatly optimizing my setting and resetting my router, even replacing it after one CSR sent me a new one.  One CSR told me that since I had seven devices connected that I should expect 1/7th performance.  (I nearly just hung up on him.)  Another CSR said they only guarantee performance on the VZ network and since their speedtest was fine as far as they were concerned there wasn't a problem.

I'm getting a little frustrated.  Appreciate any advice on what's going on here and how I can get escalated.

Matt

[Hubrisnxs]

Hi there,

     The Verizon Speedtest keeps all traffic for the speedtest on the Verizon Network.  That means that it is a good isolation tool.  It tells you IF the problem really is on the verizon network or outside of the verizon network.  If it's testing good at the verizon network, then you can move onto possible other sources of trouble.  

a couple things you should be doing to isolate the problem, is make sure you are wired when doing these tests. 

If you're wireless, then the problem is probably different altogether and we can probably help with that too.

The speedtest doesn't test DNS.  It MAY or MAY not have an impact, but try changing the DNS first.   Try using open DNS.

208.67.222.222

208.67.220.220

Here is a walkthrough to change your DNS settings in the router (note: verizon changed default passwords to be the serial number of your router, and it's case sensitive, and it's under the UPC BAR code on the sticker of your router. it looks like this kind of CSJx_________    )

I unfortunately can not see your images posted, but if you could post a normal traceroute to yahoo.com or google.com when you're having your slowdown that will probably help me out more.

The wired vs wireless question is very important, I may have glossed over it. 

Also if I could give a word to the wise on the speedtest.net speed tests.  A LOT of them are ran by mom and pop shops and you really don't know what kinda of bandwidth they have, or what kind of customer base they serve, so try your best to find BIG providers to do your test from. test from multiple speedtest.net servers, and verify what you are getting on multiple pops.  The verizon one should be your main test, to see if it's a problem on verizon's side or if it's a problem out on the internet, and then you should test at speedtest.net to confirm or isolate your problem further.

[mattgoff]

Thanks for the quick reply!  Sorry for not including this originally, but all of the tests in my original post were performed over a wired connection.  I have also repeated all of these test on two different computers (both wired) and the results are the same, indicating that this isn't a local machine (or virus or spyware) issue.  For 100% completeness, I ran the test on a fresh-from-install XUbuntu 11.10 Virtualbox instance I happened to have saved, same result.

I updated DNS to OpenDNS, router dropped WAN and reconnected, I ran ipconfig /flushdns, and reran the speedtest.net version of the test w/o any change: still slow d/l and regular u/l and ping.  It was worth trying, but as you said I would be surprised if this is a DNS issue based on what I'm seeing.

Not sure why the pictures didn't make it, but they just reinforced the text.  I'll try to upload again here:

The WinMTR is basically a more sohpisticated tracert (all #s are ping times and loss rates), so the earlier and below results should be similar (just repeated to get statistical data) but as requested here is a tracert to yahoo.com:

C:\>tracert yahoo.com

Tracing route to yahoo.com [209.191.122.70]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms myrouter.home [192.168.1.1]
2 7 ms 7 ms 7 ms L100.TAMPFL-VFTTP-88.verizon-gni.net [173.78.118.1]
3 14 ms 12 ms 12 ms G0-9-4-7.TAMPFL-LCR-21.verizon-gni.net [130.81.105.116]
4 13 ms 12 ms 12 ms so-5-0-2-0.LCC1-RES-BB-RTR1-RE1.verizon-gni.net [130.81.28.212]
5 44 ms 44 ms 45 ms xe-11-1-2-0.LCC2-RES-BB-RTR1-RE1.verizon-gni.net [130.81.17.192]
6 45 ms 56 ms 44 ms so-6-0-0-0.ASH-PEER-RTR1-re1.verizon-gni.net [130.81.10.90]
7 44 ms 44 ms 45 ms 130.81.14.98
8 46 ms 44 ms 44 ms ae-6.pat2.dce.yahoo.com [216.115.102.176]
9 68 ms 70 ms 66 ms ae-4.pat2.che.yahoo.com [216.115.101.145]
10 60 ms 72 ms 142 ms ae-7.pat2.dax.yahoo.com [216.115.96.60]
11 66 ms 70 ms 69 ms ae-1-d101.msr1.mud.yahoo.com [216.115.104.99]
12 67 ms 64 ms 67 ms te-8-2.fab2-a-gdc.mud.yahoo.com [209.191.78.157]

13 73 ms 70 ms 69 ms te-9-2.bas-c1.mud.yahoo.com [209.191.78.169]
14 73 ms 64 ms 67 ms ir1.fp.vip.mud.yahoo.com [209.191.122.70]

I'm not sure I 100% agree with your statement about the Verizon Speedtest isloating the problem.  All the test could do would be to confirm that the problem is inside VZ's network (if the result of the test shows slowness).  Here's a hypothetical: since my Verizon Speedtest is normal but speedtest.net is not, and I'm showing lost packets and wildly variable performance to multiple Tier 1 locations, it could be there is a configuration problem on VZ's side that is affecting the routing of inbound packets.

Since the issue is identical for various hosts in different data centers and with different bandwidth providers, it would seem that this is indeed a VZ problem.  For example, I happen to know that CBSSports.com is hosted at Terremark in Miami, and Terremark is also their bandwidth provider.  This is relatively close to me (Tampa region), so it's a good short range test.  Here are the results:

WinMTR statistics

Host	% Lost	Sent	Recv	Best	Avrg	Wrst	Last
myrouter.home	0	167	167	0	0	6	0
L100.TAMPFL-VFTTP-88.verizon-gni.net	0	167	167	5	8	22	7
G0-9-4-7.TAMPFL-LCR-22.verizon-gni.net	9	123	113	10	13	39	13
so-2-0-0-0.TPA01-BB-RTR2.verizon-gni.net	6	136	129	0	19	148	12
0.ge-3-2-0.XL4.MIA4.ALTER.NET	6	136	129	15	21	86	17
0.xe-8-0-0.XL2.MIA19.ALTER.NET	5	140	134	16	26	120	20
0.xe-11-3-0.GW1.MIA19.ALTER.NET	1	163	162	18	24	99	20
terremark-gw.customer.alter.net	8	127	118	23	25	39	25
t8-1.gw1.mia.terremark.net	0	167	167	23	31	233	28
66.175.114.250	7	131	123	31	34	51	35
cbs-vweb2.tm.cbsig.net	0	167	167	22	26	34	25

Compare this to Last.fm which is in a data center in London with bandwidth provided by Level3: 

WinMTR statistics

Host	% Lost	Sent	Recv	Best	Avrg	Wrst	Last
myrouter.home	0	102	102	0	0	1	0
L100.TAMPFL-VFTTP-88.verizon-gni.net	0	102	102	6	8	14	8
G0-9-4-7.TAMPFL-LCR-22.verizon-gni.net	8	75	69	10	14	46	13
so-4-0-0-0.TPA01-BB-RTR2.verizon-gni.net	13	66	58	11	17	61	12
so-4-1-0-0.ATL01-BB-RTR2.verizon-gni.net	10	72	65	23	29	58	25
0.ae4.BR3.ATL4.ALTER.NET	0	102	102	23	29	88	25
xe-11-3-0.edge4.Atlanta2.Level3.net	9	73	67	0	28	86	28
vlan51.ebr1.Atlanta2.Level3.net	0	102	102	26	31	42	34
ae-6-6.ebr1.Washington12.Level3.net	0	102	102	41	45	57	44
ae-1-100.ebr2.Washington12.Level3.net	10	71	64	42	48	51	50
4.69.148.49	8	76	70	50	52	61	53
ae-42-42.ebr2.London1.Level3.net	0	102	102	115	118	122	118
ae-57-222.csw2.London1.Level3.net	5	84	80	116	119	123	120
ae-24-52.car3.London1.Level3.net	13	66	58	118	126	264	123
195.50.90.182	7	81	76	0	121	280	115
playground.last.fm	4	89	86	115	118	124	123

So, two very different physical destinations, routes, hosting providers, bandwidth providers, and the results are the same.  Wouldn't you agree that it appears that, despite the normal Verizon Speedtest result, it's like that the problem is still on VZ's end?

[Hubrisnxs]

It's still possible that it's on the verizon side, but from both the traceroute and the vz speedtest result I can't feel 100% certain.

I wonder what your real world download test is.  maybe go to download.microsoft.com and download a large file (ages of empire is a big file)    you don't have to let it complete if you don't want to, or you can dump it afterwards but it should give you a good indicator of what your real world download looks like.   It probably downloads bits vs bytes, so you may have to multiply what you see X 8 to get the right conversion for speed. 

but give that a shot and lets see what it really looks like.     Also if it's REAL slow still at microsofts download, then download the same file twice, simultaneously.   If your download gets cut in half from doing both downloads, then its probably a receive buffer registry entry that needs to be tweaked.    you can also check your receive buffer setting by doing the VZ speedtest, and after it's done you can click analysis and paste those results here.  you mentioned you have already tweaked using the vz optimizer, but sometimes there are inconsistincies in how it does the reg edit.   so it may or may not have took, or you may have conflicting reg entries.  

Those would be done just to isolate further that it is not your PC, and that it is more an outside issue.    also maybe post what city you are in or geographic location, and others can say if they are or are not in the same area, and experiencing the same thing.    Looks like you said you are in the TAMPA FL area so hopefully some local users can chime in with how they look on speed right now

[mattgoff]

I'm in Bradenton, FL (about an hour south of Tampa).

I pulled up the .NET 2.0 SP2 page at Microsoft and began a simultaneous download of the two bigger downloads, 40-50MB each.  Shockingly, the first time I ran the test, they blazed in!  I reran Speedtest.net and it reported 15 Mbps down!  Sadly, I tried doing some 1080p YouTube streaming to further test, and while it started fine it soon started stuttering.  I went back to Speedtest.net and I'm back to ~1Mbps performance.  Rerunning the multiple download test put me back at a crawl.  Results below.

15 Mbps = a little less than 2M byes/sec.  I'm getting an average of 67k bytes/sec perfmon and Chrome corroborate (as does Task Manager, less conveniently).

If I add a 3rd or 4th file, no change to aggregate download speed.

If I run verizon.net/speedtest, I get the following (shows max at the correct 2M bytes/sec:

Checking for Middleboxes . . . . . . . . . . . . . . . . . . Done
SendBufferSize set to [261360]
running 10s outbound test (client to server) . . . . . 5.39Mb/s
running 10s inbound test (server to client) . . . . . . 15.68Mb/s

------ Client System Details ------
OS data: Name = Windows XP, Architecture = x86, Version = 5.1
Java data: Vendor = Sun Microsystems Inc., Version = 1.6.0_31

------ Web100 Detailed Analysis ------
Client Receive Window detected at 1045440 bytes.
10 Mbps Ethernet link found.
Link set to Full Duplex mode
No network congestion discovered.
Good network cable(s) found
Normal duplex operation found.

Web100 reports the Round trip time = 506.11 msec; the Packet size = 1452 Bytes; and 
No packet loss was observed.
This connection is receiver limited 25.62% of the time.
This connection is sender limited 71.3% of the time.
This connection is network limited 3.07% of the time.

Web100 reports TCP negotiated the optional Performance Settings to: 
RFC 2018 Selective Acknowledgment: ON
RFC 896 Nagle Algorithm: ON
RFC 3168 Explicit Congestion Notification: OFF
RFC 1323 Time Stamping: OFF
RFC 1323 Window Scaling: ON
Information: Network Middlebox is modifying MSS variable
Server IP addresses are preserved End-to-End
Information: Network Address Translation (NAT) box is modifying the Client's IP address
Server says [96.254.26.87] but Client says [10.17.252.13]

Just restarted the Microsoft downloads and performance is back to 15Mbit.  Pause them a Speedtest shows full speed again.  If I stream YouTube 1080p, it stutters, back to Speedtest back to 1Mbps and MS downloads restarted are slow again.

[Hubrisnxs]

well your send buffer is perfect, and you're full duplex, so pc is out, but now you mention something that may be the crux of your problem.  is it pretty much google and youtube that you are seeing the slow downs,   or is it truly more than those core sites and their slow down.

The reason why I ask is because youtube and google are having some pretty nasty routing issues specifically, so if browsing was normal in most other places - but when doing google youtube transactions, then that may be the problem 

[mattgoff]

I was just using YouTube as a quick test since they have a convenient high-bitrate stream.  I'm seeing slowdowns nearly universally.

Last night at around10pm, things were almost back to normal, consistent 15+ Mbps speedtests.  This morning, back to reduced performance again, in fact even worse, running at around 756kbps down (still 5Mbps up).  It's so bad this morning that I feel like I'm back to DSL days-- I'm literally seeing images render one-by-one when I load sites.

So, how do I get VZ to help me?  Other than continuing to call L1 and hope they eventually escalate me, I'm at a loss.  It's like they don't look at my account history-- every time I call I go through the same questions, the same diagnostics, the same claim that there isn't a problem, and the eventual offer to ship me a new modem (took them up on it once, no effect).  It doesn't matter how compentent you are or how much evidence you have that there really is a problem with your service.

If I can't get them to fix this, I'm going to have to jump to Brighthouse.  I've had FiOS for three years (so I'm no longer under contract) and have been really happy, so I'd rather just get it fixed....  seems like I just need to break through the L1 barrier to someone who can do more than the checklist.

[Hubrisnxs]

well if level 1 support is giving you the run around, then try to go to www.twitter.com/verizonsupport    those guys are particularly good, and tend to be level 2 technicians or at least have very good access to level 2 technicians.   You might have to DM to get them to send you a secure form.     It's a round about way of getting to level 2, but the twitter feed doesn't like bad press, and if level 1 is inattentive then that might be a good way to get them. 

OR you can try level 1 support again, and make sure to call when the problem is REAL bad,  make sure you are hardwired, and you have your ducks in a row.   It's a mixed lot when you call into level 1 support, some guys are very good, some not so much, but if you're patient and can let them get to the end of the road with their troubleshooting, and have them agree that sending you multiple modems is a bad idea, then they may either send you a technician (who will work with level 2 technicians) or they will reach out to one themselves.  

I personally like the twitter support, and the online chat - but you definetely have to play the game with online chat and calling into the call center.  the twitter support seems to be quicker for bigger issues and more responsive to me at least.

[mattgoff]

Had that same thought about an hour ago.  Posted my plea, linked to this thread, hopefully they'll get me to the right person.  In the meantime I may continue L1 roulette.  Definitely agree that there's a huge quality difference-- some CSRs are clearly stuck on the script.  Not that I blame them, everyone has to start somewhere, I just wish that the support process includes reading the notes on my account (especially ones from Friday) so we can continue from where we left off rather than asking me to reset the router for the 5th time.

Thanks for all your advice so far.  Good to have someone to bounce what I've done off of to make sure I'm not missing anything.

[Hubrisnxs]

just be mindful that there is acceptable packet loss, and unacceptable packet loss, and the stuff that appears in the VERIZON routers, and the alter.net look acceptable, plus they are passing the traffic.    the traffic being passed does help show that there is at least a peering point and that the data goes through one company's network to another.   

in addition to talking to verizon, try to do a who is look up for the other rotuers that are dropping your packets, and then send their noc/admin an email and see if they will look 

definetely keep us updated as to the progress though, it might be helpful for others if they have a similiar problem in the future.