Extremely poor performance to various websites during peak hours - peering issue?
mthome1056
Enthusiast - Level 1

Since approximately 26 Jan 2015, we have been experiencing extremely poor performance to several specific websites during peak hours, 9pm-11pm.  Page loads take many minutes (or never complete), etc.  Many other sites are perfectly fine, and if I connect to the same sites at the same time through my work VPN, they perform extremly well (under 1sec).  There is no correlation between OS or browser and the poor performance (tested on Windows, OSX, Linux) and speedtest typically reports over 80/80.

I've not caught any packet loss using periodic pings.  Traceroutes indicate a bunch of different routes being used (I assume this is expected for CDNs), but here's one example of an obviously bad single traceroute:

$ traceroute images.dailykos.com
traceroute to fallback.global-ssl.fastly.net (23.235.46.185), 64 hops max
1 10.1.1.1 0.464ms 0.364ms 0.298ms
2 173.48.161.1 5.223ms 4.793ms 4.904ms
3 130.81.191.250 7.387ms 7.466ms 7.366ms
4 130.81.151.76 4.909ms 4.905ms 4.932ms
5 152.63.20.117 12.441ms 12.433ms 12.434ms
6 152.63.4.141 12.373ms 12.468ms 12.370ms
7 152.179.220.126 14.989ms 14.937ms 14.872ms
8 68.86.84.249 12.505ms 12.259ms 14.988ms
9 68.86.85.25 22.409ms 22.433ms 22.519ms
10 68.86.85.1 19.817ms 19.894ms 19.919ms
11 68.86.83.74 17.389ms 19.870ms 22.460ms
12 4.53.116.18 24.815ms 25.080ms 24.851ms
13 * 208.122.44.198 2849.473ms 29.832ms
14 173.231.134.18 22.430ms * 2026.736ms

The sites in question use the Fastly CDN - my theory is that the VZ-Fastly connection is under provisioned or is broken, but I'm out of my depth at this point.  

0 Likes
Re: Extremely poor performance to various websites during peak hours - peering issue?
Not applicable

"This is a nationwide peering/routing issue that was intentionally created by Verizon to create congestion and (1) extract priority fees from the likes of Netflix and (2) get customers to subscribe to higher speed tiers. This has been acknowledged by the FCC. Verizon is building new interconnections to remedy the problem, but it will persist for at least another 6 months."

Re: Extremely poor performance to various websites during peak hours - peering issue?
neoark
Enthusiast - Level 1

Where are you quoting this from? I am also experiecing slow down from 7-10 PM especially traffic from EU.

0 Likes