- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
i have fios in pennsylvania. for several months, i've been observing a problem where many google / youtube servers are slow / stuttering in response. it seems to be getting worse.
the problem is easiest to notice with youtube videos. from the user's perspective, sometimes i can't even play a 240p video without it buffering. sometimes reloading the page makes it load fully almost instantly; sometimes changing (even *increasing*) the resolution makes it load instantly. the problem doesn't seem to depend on the time of day.
i have found many other threads on this and other forums (youtube support, dslreports) complaining about consistent youtube trouble over fios, so i think this problem might affect a lot of users.
from my perspective over the network, it's pretty obvious to me that the problem is a certain set of routers under verizon's control which border google's network.
here's my research:
at the network level, there seems to be severe (north of 40%!) packetloss to all hosts behind this router.
here's a youtube content host for a video i just watched:
steve-local@server:~$ host v16.lscache2.c.youtube.com
v16.lscache2.c.youtube.com has address 173.194.8.38
steve-local@server:~$ sudo ping -f -c 100 173.194.8.38
PING 173.194.8.38 (173.194.8.38) 56(84) bytes of data.
...............................................
--- 173.194.8.38 ping statistics ---
100 packets transmitted, 53 received, 47% packet loss, time 1398ms
rtt min/avg/max/mdev = 17.373/20.978/23.746/1.382 ms, pipe 3, ipg/ewma 14.127/20.853 ms
much of google seems afflicted:
steve-local@server:~$ sudo ping -f -c 100 www.google.com
PING www.l.google.com (72.14.204.103) 56(84) bytes of data.
............................................
--- www.l.google.com ping statistics ---
100 packets transmitted, 56 received, 44% packet loss, time 1436ms
rtt min/avg/max/mdev = 17.329/20.117/22.279/1.221 ms, pipe 2, ipg/ewma 14.507/19.689 ms
this amount of packetloss is totally unacceptable.
here's my traceroute to that content host:
steve-local@server:~$ traceroute 173.194.8.38
traceroute to 173.194.8.38 (173.194.8.38), 30 hops max, 60 byte packets
1 10.92.1.1 (10.92.1.1) 0.295 ms 0.286 ms 0.307 ms
2 L100.PITBPA-VFTTP-11.verizon-gni.net (72.95.222.1) 7.894 ms 7.948 ms 7.923 ms
3 G12-0-0-447.PITBPA-LCR-04.verizon-gni.net (130.81.180.36) 8.034 ms 8.020 ms 8.011 ms
4 P5-0-0.PITBPA-LCR-03.verizon-gni.net (130.81.27.250) 15.034 ms 17.704 ms 17.679 ms
5 so-15-3-0-0.LCC1-RES-BB-RTR1-RE1.verizon-gni.net (130.81.28.204) 17.537 ms 17.650 ms 17.665 ms
6 0.so-2-3-0.XL3.IAD8.ALTER.NET (152.63.30.17) 17.899 ms 0.so-2-1-0.XL3.IAD8.ALTER.NET (152.63.30.2) 14.760 ms 0.so-1-2-0.XL3.IAD8.ALTER.NET (152.63.37.117) 14.821 ms
7 TenGigE0-6-1-0.GW7.IAD8.ALTER.NET (152.63.32.194) 19.366 ms 20.904 ms TenGigE0-4-0-1.GW7.IAD8.ALTER.NET (152.63.37.154) 19.827 ms
8 google-gw.customer.alter.net (152.179.50.106) 37.220 ms 37.202 ms 37.190 ms
9 216.239.46.250 (216.239.46.250) 19.648 ms 91.456 ms 216.239.46.248 (216.239.46.248) 14.902 ms
10 216.239.48.113 (216.239.48.113) 21.163 ms 21.098 ms 216.239.48.109 (216.239.48.109) 75.934 ms
11 66.249.95.143 (66.249.95.143) 15.683 ms 18.064 ms 18.093 ms
12 173.194.8.38 (173.194.8.38) 20.528 ms 21.418 ms 22.429 ms
hop #8 (152.179.50.106) is where the problems start for me. all the earlier alter.net hops return pings with no loss, but i experience the same 40%+ loss to 152.179.50.106 and all hops afterward.
the bogus name "google-gw.customer.alter.net" does not resolve.
some other problematic google hosts show 152.179.50.62 (also with bogus name google-gw.customer.alter.net) as the problematic hop to google, but is otherwise similar. 152.179.50.106 and 152.179.50.62 are the only two i have seen so far.
there are some google hosts that also go through alter.net that have no problems at all (note that www.youtube.com just serves youtube's ui, not the videos themselves):
steve-local@server:~$ host www.youtube.com
www.youtube.com is an alias for youtube-ui.l.google.com.
youtube-ui.l.google.com has address 64.233.169.93
youtube-ui.l.google.com has address 64.233.169.136
youtube-ui.l.google.com has address 64.233.169.190
youtube-ui.l.google.com has address 64.233.169.91
steve-local@server:~$ sudo ping -f -c 100 64.233.169.91
PING 64.233.169.91 (64.233.169.91) 56(84) bytes of data.
--- 64.233.169.91 ping statistics ---
100 packets transmitted, 100 received, 0% packet loss, time 1307ms
rtt min/avg/max/mdev = 19.909/22.841/28.183/1.482 ms, pipe 3, ipg/ewma 13.204/23.052 ms
steve-local@server:~$ traceroute 64.233.169.91
traceroute to 64.233.169.91 (64.233.169.91), 30 hops max, 60 byte packets
1 10.92.1.1 (10.92.1.1) 0.286 ms 0.282 ms 0.302 ms
2 L100.PITBPA-VFTTP-11.verizon-gni.net (72.95.222.1) 8.147 ms 8.144 ms 8.135 ms
3 G12-0-0-347.PITBPA-LCR-03.verizon-gni.net (130.81.180.34) 10.521 ms 10.571 ms 10.546 ms
4 so-15-3-0-0.LCC1-RES-BB-RTR1-RE1.verizon-gni.net (130.81.28.204) 17.800 ms 17.855 ms 17.830 ms
5 0.so-2-3-0.XL3.IAD8.ALTER.NET (152.63.30.17) 20.314 ms 0.xe-4-1-0.XL3.IAD8.ALTER.NET (152.63.2.249) 17.850 ms 0.so-2-3-0.XL3.IAD8.ALTER.NET (152.63.30.17) 20.334 ms
6 0.so-3-2-0.XL3.NYC4.ALTER.NET (152.63.0.182) 25.099 ms 27.268 ms 0.xe-1-1-1.XL3.NYC4.ALTER.NET (152.63.1.146) 24.744 ms
7 TenGigE0-4-2-0.GW8.NYC5.ALTER.NET (152.63.16.73) 27.278 ms TenGigE0-6-4-0.GW8.NYC4.ALTER.NET (152.63.21.121) 63.215 ms TenGigE0-6-0-0.GW8.NYC4.ALTER.NET (152.63.22.41) 60.751 ms
8 Internet-gw.customer.alter.net (152.179.72.66) 23.130 ms 21.442 ms 21.323 ms
9 216.239.43.114 (216.239.43.114) 28.817 ms 209.85.252.215 (209.85.252.215) 23.902 ms 23.769 ms
10 72.14.239.93 (72.14.239.93) 38.671 ms 43.725 ms 40.372 ms
11 64.233.175.109 (64.233.175.109) 24.275 ms 21.282 ms 22.422 ms
12 72.14.232.21 (72.14.232.21) 33.984 ms 216.239.49.145 (216.239.49.145) 36.295 ms 36.274 ms
13 yo-in-f91.1e100.net (64.233.169.91) 21.187 ms 21.325 ms 21.154 ms
this host goes through alter.net, but hits a different border gateway (152.179.72.66, Internet-gw.customer.alter.net) on the way to google.
all google-owned sites aggressively use dns-based localization, so www.google.com doesn't always resolve to a host on the other side of the problem routers. this explains why refreshing the page sometimes helps, as a slightly different address can make all the difference.
users can sort of work around the problem by using a different public dns server (8.8.8.8 and 8.8.4.4; 4.2.2.1 through 4.2.2.6; or 74.82.42.42). if you're lucky, you'll get a favorable address for the host you're trying to get to.
the addresses of the problem routers, 152.179.50.106 and 152.179.50.62, are owned by verizon business (http://bgp.he.net/ip/152.179.50.106 , http://bgp.he.net/AS701, http://en.wikipedia.org/wiki/UUNET). i'm only a networking hobbyist, but i don't see any way that that this can be anyone's responsibility other than verizon's.
so please fix it.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
mtr shows a concise, visual view of the problem:
root@router:~# mtr -4 --report -w -c 100 www.google.com HOST: router Loss% Snt Last Avg Best Wrst StDev 1.|-- L100.PITBPA-VFTTP-11.verizon-gni.net 0.0% 100 8.8 6.2 3.8 9.7 1.3 2.|-- G12-0-0-347.PITBPA-LCR-03.verizon-gni.net 0.0% 100 8.5 7.3 4.0 10.9 1.3 3.|-- so-15-3-0-0.LCC1-RES-BB-RTR1-RE1.verizon-gni.net 1.0% 100 11.8 16.4 11.0 104.4 14.2 4.|-- 0.so-2-1-0.XL3.IAD8.ALTER.NET 0.0% 100 13.1 20.2 12.3 134.9 17.5 | `|-- 152.63.30.17 | |-- 152.63.36.233 | |-- 152.63.32.133 | |-- 152.63.2.249 | |-- 152.63.37.117 | |-- 152.63.34.201 5.|-- TenGigE0-4-0-1.GW7.IAD8.ALTER.NET 0.0% 100 14.5 17.1 12.4 31.8 2.6 | `|-- 152.63.37.78 | |-- 152.63.37.82 | |-- 152.63.32.190 | |-- 152.63.32.194 | |-- 152.63.37.150 6.|-- google-gw.customer.alter.net 47.0% 100 53.2 51.5 43.5 101.8 8.5 7.|-- 216.239.46.248 54.0% 100 16.9 20.9 13.5 139.6 22.1 8.|-- 66.249.94.54 53.0% 100 20.7 22.6 14.0 32.6 5.0 | `|-- 66.249.94.46 9.|-- iad04s01-in-f103.1e100.net 45.0% 100 24.3 20.5 17.7 25.6 1.7
what should i conclude here? is fios' network failing? is fios / verizon doomed? why can't i watch a youtube video without verizon losing half of it?
perhaps i should report the imminent collapse of the fios network to my favorite tech blog.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have to go back and try to find the article, but I read that Google is aware of the problem. It actually affects more than just Verizon customers. Many ISP's are having the same problem. I was reading that the problem is that many of the Google gateways were overloaded with too many connections. So they were working with Verizon and other ISP's to add additional gateway connections and bandwidth. To be honest, I just skimmed over it as it really has never affected my connnection. Youtube always plays smoothly to me. But if I remember right it, the article was saying that it had a timeline saying about 3rd Qtr of 2011. If I can find it, I'll post a link.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I was doing some troubleshooting on my connection tonight and found this exact same problem: specifically to 152.179.50.62 google-gw.customer.alter.net, in my case.
Alter.net used to be UUnet, MCI/Worldcom, and is now owned by Verizon. This is solidly a Verizon issue.
I wish we knew how to reach out to the right people from this.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
unfortunately, the very fact that you don't have any trouble underscores verizon's need to respond here. traffic is getting from verizon to google with no problem in some circumstances / for some customers, but not all.
in any case, i hope you are wrong about the 3q 2011 date. sustained 50% loss to a major content network is unacceptable to begin with; a multiple-month turnaround time for a fix makes me think about demanding reimbursement.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@sbrudenell wrote:makes me think about demanding reimbursement.
From whom? youtube is a free service. The MTR you posted shows there is nothing wrong with your FIOS connection. All the packet loss is occurring at the boundary to google's network (node 6) and within google's network (nodes 7 - 9).
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
152.179.50.106 and it's related google-gw.customer.alter.net (this name has many different IPs) hops have been problematic for two and a half years for me. I've got a monitor set up both from home and from my web server. The packet loss you've noticed doesn't seem to really occur so much coming from my web server, however it does seem the trend for loss coming from my connection occurs mostly during the Winter and Summer months, mainly during the evening. At the same time I also experience slow speeds to many YouTube video servers. Google websites, excluding Google Maps, not so much. Other ISPs in my area do see slow-downs to YouTube, but maybe at different times of the year. They do tend to fair better than Verizon during certain times of the day though, often achieving higher speeds and better latency.
I will say though that the DSL certainly does feel faster during the very early morning hours. Just seems to have better response times to sites despite no variation in latency. Speeds and jitter remain the same the whole time though. Right now YouTube and the Google sites are working fine for me, despite the packet loss coming from that hop and sequential hops.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
to your post: I do find different isps have different amounts of loss to the same google servers. but in all my testing so far, fios has been consistently the worst.
I have been running a smokeping test from dslreports to the problem router all day. so far it seems to have the same 50% loss as I see from over the fios connection itself. I'll continue to monitor it.
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
There you go. Feel free to mark the post you feel best solves your issue.