Can Verizon please stop messing with packets across the Atlantic?

[chrisjbuckley]

Verizon has been interferring with translantic traffic since 2014-04-22.  Detailed below are two ping graphs that detail both daily and monthly graphs. You will see ping response times were <100ms up until 22/4. From 22/4 - present my ping times to London are shaped from around 10:00-23:59 EDT. 

When is this going to be fixed? 

[eljefe2]

Your graphs don't show any evidence that the changes in latency are happening within Verizon's network.

I too see variations in latency to various servers in Europe but those variations are usually happening on the other side of the Atlantic where Verizon has no control.

You might try running PingPlotter, rather than the prgram you used to provide those graphs, to see where the delays are actually occurring.

And having said that, it amazes me that Internet connections across the Atlantic work as well as they do.  I think we've become spoiled in expecting packets running under the ocean, and then through many servers on the other side, to be as reliable as turning on a light bulb in our home.  Me,  I've come to expect and accept some variation in perfomance.

[chrisjbuckley]

A few points.

1. My Comcast business connection experiences none of the latency issues I am experiencing with my FIOS connection.

2. I'm entirely prepared to accept this is an issue with one of their transatlantic peers, but my contract is with Verizon and I would like them to resolve it. If it's affecting me, it's going to be affecting many others also.

3. Isn't it funny that suddenly at 23:59 my connection miracously recovers from the high latency experienced? This is consistent behaviour, *every single day* since 22/4. 

4. Being from Britain, I'm used to ISPs interferring with traffic in as many ways possible. The symptoms i'm experiencing are exactly similar to those I would experience when my connection was shaped by my UK ISP. 

5. The problem is also replicated to many other UK based IPs. 

[smith6612]

---

@chrisjbuckley wrote:

A few points.

3. Isn't it funny that suddenly at 23:59 my connection miracously recovers from the high latency experienced? This is consistent behaviour, *every single day* since 22/4. 

---

I can confirm this observation. Tonight, my game running over the latent points on Level3 magically improved at 23:50 almost immediately.

[eljefe2]

Interesting...but the question in my mind remains who's throwing the switch at  ~midnight?  Does Verizon have any control over that?

[smith6612]

I wouldn't know. It's possible that Verizon has such an odd case of network usage, it could be just enough people calling it quits around that time to cut down on a massive buffer of data being queued up between Level3 and Verizon. We'd need to get someone in-the-know from both networks to explain what is up. 100ms of latency on 10Gbps+ circuits isn't out of the ballpark for extreme cases of saturation, especially if the gear is holding a buffer of data. Normally such circuits go to roughly 10ms when they're full (you can usually see this effect when you max your FiOS connection out. FiOS goes to +10ms latency).

Any system that would be implemented in the event that this is shaping and throttling would have a time of day , or a usage scenario aspect to it. I'm sure the providers in the UK do something similar. It would be too much work (and thus much too expensive) to manually flip a switch than to just let your gear do it for you. Sandvine for the longest time here in the US used to do something similar back when Comcast used them. If you didn't pay for a business circuit, you could safely say that if you had certain traffic patterns, a system would automatically nail you with a throttle without any human intervention.

Even then, there is no excuse if this is a capacity issue not to be beefing up capacity. You don't go out and buy extremely expensive backbone rounting equipment, blades, and line cards and forget to build in future proofing. For what it's worth, anything unused is just a depreciating asset in simple accounting.

[eljefe2]

When I said "flip the switch" I of course meant who configured the system that automatically makes the change around midnight.  I wasn't suggesting that someone actually moved a mechanical switch.

As for finding someone in-the-know from both networks, I'll be staying tuned to read about it if and when that happens.

In the meantime, it's the vagaries of the Internet to me.   I'm still hard pressed to understand how any ISP an individual uses can be responsible for that individual's packets regardless of what corner of the globe the packets are traveling to and from.

[smith6612]

es

---

@eljefe wrote:

When I said "flip the switch" I of course meant who configured the system that automatically makes the change around midnight.  I wasn't suggesting that someone actually moved a mechanical switch.

As for finding someone in-the-know from both networks, I'll be staying tuned to read about it if and when that happens.

In the meantime, it's the vagaries of the Internet to me.   I'm still hard pressed to understand how any ISP an individual uses can be responsible for that individual's packets regardless of what corner of the globe the packets are traveling to and from.

---

Gotcha. Just wanted to be sure 🙂

I can see what I can dig up. I have an idea here of what may be up here. Would you be able to supply a traceroute for me? I wonder if you're hitting Level3, which has been a big problem for me as of late. Shoud have asked for that earlier before blaming the Level3 issues. Tracing the site you're pinging here, I'm getting a peering point between Verizon and Global Crossing before handing over to Jump Networks. I can't see the return path.

If the site is on your own VPS, that may become helfpul.

[eljefe2]

Don't know if you were asking me for a traceroute, or the original poster, but, FWIW, here's a PingPlot of a server I use in Denmark.

No Level3 involved.

If it will help, you could give me the IP of the server you're having the problem with and I can see what my routing is to it.

[chrisjbuckley]

Level 3 don't appear to be handling any of my packets. This is to a VPS in London, UK.  I can replicate this behaviour against nearly every other UK based IP I ping. 

traceroute to cjbuckley.net (85.119.83.54), 30 hops max, 40 byte packets
1 router.home.cjbuckley.com (10.10.99.1) 0.413 ms 0.516 ms 0.670 ms
2 L100.BSTNMA-VFTTP-75.verizon-gni.net (173.48.181.1) 4.906 ms 4.919 ms 4.903 ms
3 G0-15-0-0.BSTNMA-LCR-21.verizon-gni.net (100.41.203.220) 9.238 ms 9.336 ms 9.400 ms
4 ae9-0.BOS-BB-RTR1.verizon-gni.net (130.81.163.164) 24.267 ms ae1-0.BOS-BB-RTR1.verizon-gni.net (130.81.151.60) 24.363 ms so-3-0-0-0.BOS-BB-RTR1.verizon-gni.net (130.81.151.218) 9.411 ms
5 0.ae11.XL3.NYC1.ALTER.NET (152.63.20.69) 19.028 ms 19.049 ms 18.556 ms
6 0.xe-10-0-0.BR1.NYC1.ALTER.NET (152.63.19.201) 18.764 ms 0.xe-8-0-0.BR1.NYC1.ALTER.NET (152.63.16.65) 18.806 ms 18.855 ms

Interestingly, at 15:30 yesterday (2014-05-02) my latency returned to <100ms ping times, instead of the current ~200ms ones I have been experiencing for the last week. 

To the poster who stated that this in some way isn't Verizon's fault. Sorry, but I completely disagree. My (our?) connections are clearly being shaped. If they weren't, one wouldn't see the rigid uniformity in time periods when icmp packets get mangled. Running IPSEC VPNs across the translantic, whilst these continue to operate, I do notice the connection slows when the latency creeps in, hence my desire to have it resolved.