NY fios connection speeds / routing
rbro
Enthusiast - Level 1

Hello,

I realize the following may not be directly related to FIOS, but I was hoping to get assistance with it.

I live in New York, NY and have FIOS 50/25.  During the day, I have a very fast download time of 3-4 MB/s to the URL http://speedtest-ny1.digitalocean.com/10mb.test, but at night, the connection slows down greatly to only 100-200 KB/s.  I have been able to reproduce this slowdown the past couple nights.  I tried the URL at night from another location outside of FIOS and it was very fast, so it seems related to the path my connection is taking.  I have asked the webhost Digital Ocean to check into it on their side as well.  I have extremely fast connections to their other data centers (see list at the bottom of https://www.digitalocean.com/faq).

I have included a traceroute below.  Does anyone have ideas on how I can best communicate this issue to Verizon, if it's something that they could help resolve?

I have also noticed similar slowness in the speed test at http://speakeasy.net/speedtest for the New York, NY location.  During the day, it's extremely high, close to 50Mbps, but at night, it slows down to 6-7 Mbps.  In that speed test, I actually have much faster speeds to other cities than New York, NY such as Atlanta (58 Mbps).

Please let me know if there is any other information I can provide to help narrow this down.

Thank you for your help.

Ryan

traceroute to speedtest-ny1.digitalocean.com (198.211.116.146), 30 hops max, 60 byte packets
1 Wireless_Broadband_Router.home (192.168.1.1) 2.084 ms 2.077 ms 2.072 ms
2 L100.NYCMNY-VFTTP-210.verizon-gni.net (173.56.9.1) 7.063 ms 7.070 ms 7.720 ms
3 G1-14-2-1.NYCMNY-LCR-21.verizon-gni.net (130.81.219.54) 9.895 ms 13.490 ms 13.483 ms
4 130.81.151.228 (130.81.151.228) 46.241 ms 46.243 ms 46.239 ms
5 0.xe-9-0-0.BR3.NYC4.ALTER.NET (152.63.23.181) 13.434 ms 13.978 ms 15.013 ms
6 te-7-3-0.edge2.NewYork2.level3.net (4.68.111.137) 113.349 ms 107.422 ms 107.408 ms
7 vlan51.ebr1.NewYork2.Level3.net (4.69.138.222) 100.354 ms 95.304 ms 95.293 ms
8 ae-48-48.ebr2.NewYork1.Level3.net (4.69.201.49) 7.707 ms 4.69.201.41 (4.69.201.41) 97.493 ms 98.774 ms
9 ae-2-2.ebr1.Newark1.Level3.net (4.69.132.98) 9.101 ms 9.785 ms 9.781 ms
10 ae-1-51.edge3.Newark1.Level3.net (4.69.156.11) 29.054 ms 28.177 ms 28.156 ms
11 SERVERSTACK.edge3.Newark1.Level3.net (4.28.6.70) 31.528 ms 31.508 ms 28.552 ms
12 nbg.speedtest (198.211.116.146) 62.239 ms 62.225 ms 62.172 ms

Re: NY fios connection speeds / routing
ibamars
Enthusiast - Level 3

HMM interesting. I live in the Albany NY area, I have no issues during peak time at all. I get a constant 58/39 on the 50/25 plan. I have read in the past that prime time will effect your internet. Used to happen to me all the time when I was on Time Warner, Maybe because NYC has so many people the servers are getting a little overloaded either on Verizons end or Digital Oceans end? Not 100% sure on that one. GL

0 Likes
Re: NY fios connection speeds / routing BAD HOP
Carlos4382
Enthusiast - Level 2
Hello there, I seem to be having a similar problem. I also live in NY (Long Island to be exact) and over the past week or so something seems to be going very wrong in a hop while connecting to my work VPN router, which is located in NYC. For example, i just ran a speed test to a server located in NY using speedtest.net and i'm getting this lovely speed with FiOS. i just took a look at the following site. according to this it is reporting some issues between traffic between Verizon and Cogent networks. http://www.internethealthreport.com/ Therefore, I think there is definetly a bad hop somewhere. Not sure how to go about reporting it.
Re: NY fios connection speeds / routing BAD HOP
Namronorman
Contributor - Level 2

Definitely a problem somewhere. I tried to download the 100 MB file and it took forever for just 1MB to finish. I try via VPN connection to a server I have in NY, and it goes at about 3MB/s.

Here's my MTR for my connection through FiOS:

WinMTR statistics
Host%SentRecvBestAvrgWrstLast
Wireless_Broadband_Router.home01041040000
L100.PITBPA-VFTTP-33.verizon-gni.net0104104692811
G0-1-0-5.PITBPA-LCR-21.verizon-gni.net01041045101413
xe-19-0-8-0.RES-BB-RTR1.verizon-gni.net0104104114424515
0.xe-3-0-0.BR2.IAD8.ALTER.NET010410412186314
ae17.edge1.washingtondc12.level3.net110110015193821
vl-3603-ve-227.ebr2.Washington12.Level3.net1101100748410486
ae-5-5.ebr2.Washington1.Level3.net010410418203131
ae-4-4.ebr2.Newark1.Level3.net010410471819489
ae-2-52.edge3.Newark1.Level3.net010410418214622
SERVERSTACK.edge3.Newark1.Level3.net589858590101101
nbg.speedtest117365969810299

I see constant problems with Cogent, Level3, NTT, and well... everything. Don't bother complaining though, it's not Verizon's responsibility to maintain a decent business relationship with other providers... right? guys?

0 Likes
Re: NY fios connection speeds / routing BAD HOP
smith6612
Community Leader
Community Leader

@Namronorman wrote:

Definitely a problem somewhere. I tried to download the 100 MB file and it took forever for just 1MB to finish. I try via VPN connection to a server I have in NY, and it goes at about 3MB/s.

Here's my MTR for my connection through FiOS:

WinMTR statistics
Host%SentRecvBestAvrgWrstLast
Wireless_Broadband_Router.home01041040000
L100.PITBPA-VFTTP-33.verizon-gni.net0104104692811
G0-1-0-5.PITBPA-LCR-21.verizon-gni.net01041045101413
xe-19-0-8-0.RES-BB-RTR1.verizon-gni.net0104104114424515
0.xe-3-0-0.BR2.IAD8.ALTER.NET010410412186314
ae17.edge1.washingtondc12.level3.net110110015193821
vl-3603-ve-227.ebr2.Washington12.Level3.net1101100748410486
ae-5-5.ebr2.Washington1.Level3.net010410418203131
ae-4-4.ebr2.Newark1.Level3.net010410471819489
ae-2-52.edge3.Newark1.Level3.net010410418214622
SERVERSTACK.edge3.Newark1.Level3.net589858590101101
nbg.speedtest117365969810299

I see constant problems with Cogent, Level3, NTT, and well... everything. Don't bother complaining though, it's not Verizon's responsibility to maintain a decent business relationship with other providers... right? guys?


That MTR suggests the problem is with Level3 themselves. The loss you're seeing in Washington DC is fine. The spikes in latency are also fine for as long as they are not consistently that high. Administrative protocols such as ICMP on low priority in many places. The loss you start to see after ae-2-52.edge3.Newark1.Level3.net going to Serverstack suggests a few things. Either the datacenter doesn't have enough capacity from Level3, Level3's having a problem in their network (wouldn't be surprised) connecting up to Serverstack, or there's a return path we can't see that's to blame.

The best case here is to have an admin to nbg.speedtest use that host to send an MTR back to your IP address. This will let us see the other path, where we can start to track loss a bit more.

Here's a trace from the Buffalo, NY area. Note, I am on DSL, not FiOS

Tracing route to speedtest-ny1.digitalocean.com [198.211.116.146]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  192.168.3.1
  2     7 ms     7 ms     7 ms  10.41.15.1
  3     8 ms     7 ms     7 ms  P0-0-1-2.BFLONY-LCR-21.verizon-gni.net [130.81.44.232]
  4    19 ms    18 ms    19 ms  xe-1-1-0-0.NWRK-BB-RTR1.verizon-gni.net [130.81.209.34]
  5    24 ms    23 ms    24 ms  0.xe-10-1-0.BR1.NYC1.ALTER.NET [152.63.18.225]
  6     *        *        *     Request timed out.
  7    25 ms    25 ms    24 ms  vlan51.ebr1.NewYork2.Level3.net [4.69.138.222]
  8    30 ms    30 ms    31 ms  4.69.201.45
  9    25 ms    24 ms    24 ms  ae-2-2.ebr1.Newark1.Level3.net [4.69.132.98]
 10    39 ms    40 ms    39 ms  ae-1-51.edge3.Newark1.Level3.net [4.69.156.11]
 11    45 ms     *       44 ms  SERVERSTACK.edge3.Newark1.Level3.net [4.28.6.70]

 12    74 ms    72 ms    71 ms  nbg.speedtest [198.211.116.146]

Trace complete.

WinMTR:

WinMTR Statistics

WinMTR statistics
Host%SentRecvBestAvrgWrstLast
192.168.3.101041040000
10.41.15.1010410468659
P0-0-1-2.BFLONY-LCR-21.verizon-gni.net01041046106310
xe-1-1-0-0.NWRK-BB-RTR1.verizon-gni.net0104104183612719
0.xe-10-1-0.BR1.NYC1.ALTER.NET0104104253910232
ae11.edge2.NewYork.Level3.net2553400246626
vlan51.ebr1.NewYork2.Level3.net010410424276225
4.69.201.45010410424366933
ae-2-2.ebr1.Newark1.Level3.net010410424276224
ae-1-51.edge3.Newark1.Level3.net010410427386333
SERVERSTACK.edge3.Newark1.Level3.net8807444507752
nbg.speedtest11009969748774

 I'm getting 140KB/s to your webhost. To reach my place of employment, which is a datacenter, I also go over Level3. I have no problem maxing out my connection to any public or VPN-based resources.

0 Likes
Re: NY fios connection speeds / routing
Carlos4382
Enthusiast - Level 2

I called Cogent Communications today.  They told me that they knew of a 'bandwidth issue' between hop 6 and 7 from my trace route below. The tech I called told me that the issue is being caused by high traffice and that it is planned to increase bandwidth, but he could not give me a resolution date.

I wonder if any of this is related to the ongoing dispute between Verizon / Cogent: http://publicpolicy.verizon.com/blog/entry/unbalanced-peering-and-the-real-story-behind-the-verizon-...

Tracing route to 38.108.235.162 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  192.168.1.1

  2     3 ms     3 ms     4 ms  L100.NYCMNY-VFTTP-32.verizon-gni.net [71.125.61.1]

  3     7 ms     7 ms     8 ms  G0-14-4-5.NYCMNY-LCR-22.verizon-gni.net [130.81.187.92]

  4     5 ms     5 ms     5 ms  ae0-0.NY5030-BB-RTR1.verizon-gni.net [130.81.209.118]

  5     6 ms     6 ms     6 ms  0.xe-11-3-0.BR2.NYC4.ALTER.NET [152.63.23.137]

  6    89 ms    90 ms    90 ms  204.255.168.114

  7    89 ms    78 ms    90 ms  te0-2-0-4.mpd22.jfk02.atlas.cogentco.com [154.54.46.253]

  8    85 ms    90 ms    85 ms  te3-8.mag01.jfk02.atlas.cogentco.com [154.54.27.126]

  9    90 ms    89 ms    90 ms  gi0-0-0-0.nr11.b001428-0.jfk02.atlas.cogentco.com [154.24.2.110]

10    84 ms     *       84 ms  38.108.235.162

0 Likes
Re: NY fios connection speeds / routing
Namronorman
Contributor - Level 2

Hop 6 is MCI (Verizon)... so, hopefully?

0 Likes
Re: NY fios connection speeds / routing
Buzzati
Newbie

I'm in North Jersey.  I have a similar problem with packet loss.  I've run Pingplotter consistently and it shows that I get packet loss up to 7 to 14% on peak times.  At around 4 to 6 AM in the morning the packet loss disappears.  This cycle occurs every single day, like clockwork.

 Verizon has not been able to help me fix this issue - as far as I see it's just the way the service is around the northeast.

0 Likes
Re: NY fios connection speeds / routing
rbro
Enthusiast - Level 1

I am still continuing to see slow speeds on my NYC FIOS connection at night accessing certain URL's..  During the day the download speeds are very high, but at night, accessing certain URL's is extremely slow.

Here are a few examples from tonight at 9pm Eastern in NYC:

hxxp://download.crashplan.com/installs/win/install/CrashPlan/jre/CrashPlan-x64_3.5.3_Win.exe  {Hyperlink to executable file disabled}
From a FIOS-connection, I am downloading the above at 50 KB/s.  A non-FIOS connection downloads it at 2 MB/s.

http://speedtest-ny1.digitalocean.com/10mb.test
From a FIOS-connection, I am downloading the above at 60 KB/s.  A non-FIOS connection downloads it at 5 MB/s.

http://speakeasy.net/speedtest
For New York, NY, my download speed is 2.5 MB/s.  For Atlanta, GA, my download speed is 48 MB/s.

Why is my connection so much faster to Atlanta compared to New York, even though I am physically located in New York?  Again, a non-FIOS connection has extremely fast speeds to Speakeasy's New York, NY location.  I believe there is some routing issue involved once my connection leaves the FIOS network.

Can someone from NYC on FIOS run speed tests for the above URL's to see if others are experiencing this?

Is there someone in FIOS networking that can help diagnose why certain routes leaving the FIOS network are very very slow?

Thanks for your help.

Ryan

0 Likes