google / youtube slow over fios. definitely verizon's fault (research inside), please fix
sbrudenell
Enthusiast - Level 2

i have fios in pennsylvania. for several months, i've been observing a problem where many google / youtube servers are slow / stuttering in response. it seems to be getting worse.

the problem is easiest to notice with youtube videos. from the user's perspective, sometimes i can't even play a 240p video without it buffering. sometimes reloading the page makes it load fully almost instantly; sometimes changing (even *increasing*) the resolution makes it load instantly. the problem doesn't seem to depend on the time of day.

i have found many other threads on this and other forums (youtube support, dslreports) complaining about consistent youtube trouble over fios, so i think this problem might affect a lot of users.

from my perspective over the network, it's pretty obvious to me that the problem is a certain set of routers under verizon's control which border google's network.

here's my research:


at the network level, there seems to be severe (north of 40%!) packetloss to all hosts behind this router.

here's a youtube content host for a video i just watched:


steve-local@server:~$ host v16.lscache2.c.youtube.com

v16.lscache2.c.youtube.com has address 173.194.8.38

steve-local@server:~$ sudo ping -f -c 100 173.194.8.38

PING 173.194.8.38 (173.194.8.38) 56(84) bytes of data.

...............................................

--- 173.194.8.38 ping statistics ---

100 packets transmitted, 53 received, 47% packet loss, time 1398ms

rtt min/avg/max/mdev = 17.373/20.978/23.746/1.382 ms, pipe 3, ipg/ewma 14.127/20.853 ms

much of google seems afflicted:


steve-local@server:~$ sudo ping -f -c 100 www.google.com

PING www.l.google.com (72.14.204.103) 56(84) bytes of data.

............................................

--- www.l.google.com ping statistics ---

100 packets transmitted, 56 received, 44% packet loss, time 1436ms

rtt min/avg/max/mdev = 17.329/20.117/22.279/1.221 ms, pipe 2, ipg/ewma 14.507/19.689 ms

this amount of packetloss is totally unacceptable.


here's my traceroute to that content host:


steve-local@server:~$ traceroute 173.194.8.38

traceroute to 173.194.8.38 (173.194.8.38), 30 hops max, 60 byte packets

 1  10.92.1.1 (10.92.1.1)  0.295 ms  0.286 ms  0.307 ms

 2  L100.PITBPA-VFTTP-11.verizon-gni.net (72.95.222.1)  7.894 ms  7.948 ms  7.923 ms

 3  G12-0-0-447.PITBPA-LCR-04.verizon-gni.net (130.81.180.36)  8.034 ms  8.020 ms  8.011 ms

 4  P5-0-0.PITBPA-LCR-03.verizon-gni.net (130.81.27.250)  15.034 ms  17.704 ms  17.679 ms

 5  so-15-3-0-0.LCC1-RES-BB-RTR1-RE1.verizon-gni.net (130.81.28.204)  17.537 ms  17.650 ms  17.665 ms

 6  0.so-2-3-0.XL3.IAD8.ALTER.NET (152.63.30.17)  17.899 ms 0.so-2-1-0.XL3.IAD8.ALTER.NET (152.63.30.2)  14.760 ms 0.so-1-2-0.XL3.IAD8.ALTER.NET (152.63.37.117)  14.821 ms

 7  TenGigE0-6-1-0.GW7.IAD8.ALTER.NET (152.63.32.194)  19.366 ms  20.904 ms TenGigE0-4-0-1.GW7.IAD8.ALTER.NET (152.63.37.154)  19.827 ms

 8  google-gw.customer.alter.net (152.179.50.106)  37.220 ms  37.202 ms  37.190 ms

 9  216.239.46.250 (216.239.46.250)  19.648 ms  91.456 ms 216.239.46.248 (216.239.46.248)  14.902 ms

10  216.239.48.113 (216.239.48.113)  21.163 ms  21.098 ms 216.239.48.109 (216.239.48.109)  75.934 ms

11  66.249.95.143 (66.249.95.143)  15.683 ms  18.064 ms  18.093 ms

12  173.194.8.38 (173.194.8.38)  20.528 ms  21.418 ms  22.429 ms

hop #8 (152.179.50.106) is where the problems start for me. all the earlier alter.net hops return pings with no loss, but i experience the same 40%+ loss to 152.179.50.106 and all hops afterward.

the bogus name "google-gw.customer.alter.net" does not resolve.

some other problematic google hosts show 152.179.50.62 (also with bogus name google-gw.customer.alter.net) as the problematic hop to google, but is otherwise similar. 152.179.50.106 and 152.179.50.62 are the only two i have seen so far.


there are some google hosts that also go through alter.net that have no problems at all (note that www.youtube.com just serves youtube's ui, not the videos themselves):


steve-local@server:~$ host www.youtube.com

www.youtube.com is an alias for youtube-ui.l.google.com.

youtube-ui.l.google.com has address 64.233.169.93

youtube-ui.l.google.com has address 64.233.169.136

youtube-ui.l.google.com has address 64.233.169.190

youtube-ui.l.google.com has address 64.233.169.91
steve-local@server:~$ sudo ping -f -c 100 64.233.169.91

PING 64.233.169.91 (64.233.169.91) 56(84) bytes of data.

   

--- 64.233.169.91 ping statistics ---

100 packets transmitted, 100 received, 0% packet loss, time 1307ms

rtt min/avg/max/mdev = 19.909/22.841/28.183/1.482 ms, pipe 3, ipg/ewma 13.204/23.052 ms
steve-local@server:~$ traceroute 64.233.169.91

traceroute to 64.233.169.91 (64.233.169.91), 30 hops max, 60 byte packets

 1  10.92.1.1 (10.92.1.1)  0.286 ms  0.282 ms  0.302 ms

 2  L100.PITBPA-VFTTP-11.verizon-gni.net (72.95.222.1)  8.147 ms  8.144 ms  8.135 ms

 3  G12-0-0-347.PITBPA-LCR-03.verizon-gni.net (130.81.180.34)  10.521 ms  10.571 ms  10.546 ms

 4  so-15-3-0-0.LCC1-RES-BB-RTR1-RE1.verizon-gni.net (130.81.28.204)  17.800 ms  17.855 ms  17.830 ms

 5  0.so-2-3-0.XL3.IAD8.ALTER.NET (152.63.30.17)  20.314 ms 0.xe-4-1-0.XL3.IAD8.ALTER.NET (152.63.2.249)  17.850 ms 0.so-2-3-0.XL3.IAD8.ALTER.NET (152.63.30.17)  20.334 ms

 6  0.so-3-2-0.XL3.NYC4.ALTER.NET (152.63.0.182)  25.099 ms  27.268 ms 0.xe-1-1-1.XL3.NYC4.ALTER.NET (152.63.1.146)  24.744 ms

 7  TenGigE0-4-2-0.GW8.NYC5.ALTER.NET (152.63.16.73)  27.278 ms TenGigE0-6-4-0.GW8.NYC4.ALTER.NET (152.63.21.121)  63.215 ms TenGigE0-6-0-0.GW8.NYC4.ALTER.NET (152.63.22.41)  60.751 ms

 8  Internet-gw.customer.alter.net (152.179.72.66)  23.130 ms  21.442 ms  21.323 ms

 9  216.239.43.114 (216.239.43.114)  28.817 ms 209.85.252.215 (209.85.252.215)  23.902 ms  23.769 ms

10  72.14.239.93 (72.14.239.93)  38.671 ms  43.725 ms  40.372 ms

11  64.233.175.109 (64.233.175.109)  24.275 ms  21.282 ms  22.422 ms

12  72.14.232.21 (72.14.232.21)  33.984 ms 216.239.49.145 (216.239.49.145)  36.295 ms  36.274 ms

13  yo-in-f91.1e100.net (64.233.169.91)  21.187 ms  21.325 ms  21.154 ms


this host goes through alter.net, but hits a different border gateway (152.179.72.66, Internet-gw.customer.alter.net) on the way to google.

all google-owned sites aggressively use dns-based localization, so www.google.com doesn't always resolve to a host on the other side of the problem routers. this explains why refreshing the page sometimes helps, as a slightly different address can make all the difference.

users can sort of work around the problem by using a different public dns server (8.8.8.8 and 8.8.4.4; 4.2.2.1 through 4.2.2.6; or 74.82.42.42). if you're lucky, you'll get a favorable address for the host you're trying to get to.

the addresses of the problem routers, 152.179.50.106 and 152.179.50.62, are owned by verizon business (http://bgp.he.net/ip/152.179.50.106http://bgp.he.net/AS701, http://en.wikipedia.org/wiki/UUNET). i'm only a networking hobbyist, but i don't see any way that that this can be anyone's responsibility other than verizon's.

so please fix it.

Re: google / youtube slow over fios. definitely verizon's fault (research inside), please fix
sbrudenell
Enthusiast - Level 2

mtr shows a concise, visual view of the problem:

root@router:~# mtr -4 --report -w -c 100 www.google.com  
HOST: router                                           Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- L100.PITBPA-VFTTP-11.verizon-gni.net              0.0%   100    8.8   6.2   3.8   9.7   1.3
  2.|-- G12-0-0-347.PITBPA-LCR-03.verizon-gni.net         0.0%   100    8.5   7.3   4.0  10.9   1.3
  3.|-- so-15-3-0-0.LCC1-RES-BB-RTR1-RE1.verizon-gni.net  1.0%   100   11.8  16.4  11.0 104.4  14.2
  4.|-- 0.so-2-1-0.XL3.IAD8.ALTER.NET                     0.0%   100   13.1  20.2  12.3 134.9  17.5
    |  `|-- 152.63.30.17
    |   |-- 152.63.36.233
    |   |-- 152.63.32.133
    |   |-- 152.63.2.249
    |   |-- 152.63.37.117
    |   |-- 152.63.34.201
  5.|-- TenGigE0-4-0-1.GW7.IAD8.ALTER.NET                 0.0%   100   14.5  17.1  12.4  31.8   2.6
    |  `|-- 152.63.37.78
    |   |-- 152.63.37.82
    |   |-- 152.63.32.190
    |   |-- 152.63.32.194
    |   |-- 152.63.37.150
  6.|-- google-gw.customer.alter.net                     47.0%   100   53.2  51.5  43.5 101.8   8.5
  7.|-- 216.239.46.248                                   54.0%   100   16.9  20.9  13.5 139.6  22.1
  8.|-- 66.249.94.54                                     53.0%   100   20.7  22.6  14.0  32.6   5.0
    |  `|-- 66.249.94.46
  9.|-- iad04s01-in-f103.1e100.net                       45.0%   100   24.3  20.5  17.7  25.6   1.7

 what should i conclude here? is fios' network failing? is fios / verizon doomed? why can't i watch a youtube video without verizon losing half of it?

perhaps i should report the imminent collapse of the fios network to my favorite tech blog.

0 Likes
Re: google / youtube slow over fios. definitely verizon's fault (research inside), please fix
spacedebris
Master - Level 2

I have to go back and try to find the article, but I read that Google is aware of the problem. It actually affects more than just Verizon customers. Many ISP's are having the same problem. I was reading that the problem is that many of the Google gateways were overloaded with too many connections. So they were working with Verizon and other ISP's to add additional gateway connections and bandwidth. To be honest, I just skimmed over it as it really has never affected my connnection. Youtube always plays smoothly to me. But if I remember right it, the article was saying that it had a timeline saying about 3rd Qtr of 2011. If I can find it, I'll post a link.

Re: google / youtube slow over fios. definitely verizon's fault (research inside), please fix
Icarus1
Enthusiast - Level 1

I was doing some troubleshooting on my connection tonight and found this exact same problem:  specifically to 152.179.50.62 google-gw.customer.alter.net, in my case.  

Alter.net used to be UUnet, MCI/Worldcom, and is now owned by Verizon.  This is solidly a Verizon issue. 

I wish we knew how to reach out to the right people from this. 

Re: google / youtube slow over fios. definitely verizon's fault (research inside), please fix
sbrudenell
Enthusiast - Level 2

unfortunately, the very fact that you don't have any trouble underscores verizon's need to respond here. traffic is getting from verizon to google with no problem in some circumstances / for some customers, but not all.

in any case, i hope you are wrong about the 3q 2011 date. sustained 50% loss to a major content network is unacceptable to begin with; a multiple-month turnaround time for a fix makes me think about demanding reimbursement.

0 Likes
Re: google / youtube slow over fios. definitely verizon's fault (research inside), please fix
Anti-Phish1
Master - Level 1

@sbrudenell wrote:

makes me think about demanding reimbursement.


From whom?  youtube is a free service.  The MTR you posted shows there is nothing wrong with your FIOS connection.  All the packet loss is occurring at the boundary to google's network (node 6) and within google's network (nodes 7 - 9). 

0 Likes
Re: google / youtube slow over fios. definitely verizon's fault (research inside), please fix
sbrudenell
Enthusiast - Level 2
the problem router (or at least its address) is owned by verizon, not google. the hops after the problem router all return pings just fine from any connection other than my fios. I find no evidence that there is anything wrong within google's network.
Re: google / youtube slow over fios. definitely verizon's fault (research inside), please fix
smith6612
Community Leader
Community Leader

152.179.50.106 and it's related google-gw.customer.alter.net (this name has many different IPs) hops have been problematic for two and a half years for me. I've got a monitor set up both from home and from my web server. The packet loss you've noticed doesn't seem to really occur so much coming from my web server, however it does seem the trend for loss coming from my connection occurs mostly during the Winter and Summer months, mainly during the evening. At the same time I also experience slow speeds to many YouTube video servers. Google websites, excluding Google Maps, not so much. Other ISPs in my area do see slow-downs to YouTube, but maybe at different times of the year. They do tend to fair better than Verizon during certain times of the day though, often achieving higher speeds and better latency.

I will say though that the DSL certainly does feel faster during the very early morning hours. Just seems to have better response times to sites despite no variation in latency. Speeds and jitter remain the same the whole time though. Right now YouTube and the Google sites are working fine for me, despite the packet loss coming from that hop and sequential hops.

0 Likes
Re: google / youtube slow over fios. definitely verizon's fault (research inside), please fix
sbrudenell
Enthusiast - Level 2
ugh. I didn't actually mean to make this post the accepted solution. I fat fingered the reply button, and now I can't find a way to undo this. can a mod help me here?

to your post: I do find different isps have different amounts of loss to the same google servers. but in all my testing so far, fios has been consistently the worst.

I have been running a smokeping test from dslreports to the problem router all day. so far it seems to have the same 50% loss as I see from over the fios connection itself. I'll continue to monitor it.
Re: google / youtube slow over fios. definitely verizon's fault (research inside), please fix
ElizabethS
Moderator Emeritus

There you go. Feel free to mark the post you feel best solves your issue.

0 Likes