Re: What can we do to actually fix Verizon's Netflix throttling?
Hubrisnxs
Legend

no I Actually read that.  That means that it took a different path.  That's all. 

a REAL GOOD test is if you had APPLE TV at your house, Netflix signed an agreement with apple to route those customers across a premium link.    you wouldn't have any problems if you did that.    


@TennisFreak wrote:

@Hubrisnxs

Maybe you didnt read all of my post then.


Because I clearly stated that while Netlix was not playing worth a dang on my FIOS connection I could simultaneously load Netflix via my LTE connection on my Lumia 1520 and watch the same show in full HD with no buffering.



In a little known, but public fact, anyone who is on Comcast  or Verizon and using Apple TV to stream Netflix wasn’t having quality problems. The reason for this is that Netflix is using Level 3 and Limelight to stream their content specifically to the Apple TV device. What this shows is that Netflix is the one that decides and controls how they get their content to each device and whether they do it via their own servers or a third party. Netflix decides which third party CDNs to use and when Netflix uses their own CDN, they decide whom to buy transit from, with what capacity, in what locations and how many connections they buy, from the transit provider. Netflix is the one in control of this, not Comcast or any ISP.


0 Likes
Re: What can we do to actually fix Verizon's Netflix throttling?
nick_d
Enthusiast - Level 3

@chomper87 wrote:
Verizon isn't chaning anything. Netflix has a catalog of videos. Netflix wants to stream those videos. Netflix needs to PAY for an ISP. Netflix needs to PAY for a CDN. What's so difficult to understand? If I start my own Video catalog and streaming service, should I get a free services from Verizon? Am I going to get a Free ISP? A free CDN? NO.

I'm not sure you understand how enterprise networks interconnect. Netflix IS ALREADY paying for their bandwidth from a top tier provider (at speeds greater than 1Tbps in many connection points and a fee of tens of thousands if not more). They ALREADY HAVE a CDN system using open peering. They're not trying to access the internet using bandwidth from Verizon like consumers do.  

Essentially you should think of Netflix as a really great burger place drawing tons of customers from all over. Those customers are using a highway to get to the burger place -- that highway is Verizon's network, Comcasts network, AT&Ts network, etc... a network for which the customers have already paid to access when they pay their monthy internet bill. So what verizon has done is what Chris Christie's office did in NJ -- they took their wide highway down to a lane or two and just let it back up. This is not Netflix demanding bandwidth from Verizon this is Verizon's customers demanding access to Netflix.  Netflix doesn't just send traffic without it being requested!! I don't know how that is so difficult to understand.  Verizon sees that a lot of it's customers are watching netflix and they feel like they should get more money for Netflix's success -- absolutely ridiculous.  It's like the mall seeing that a lot more people want to shop at a particular store they they try to charge that store more because they are popular - explain how that is fair.  If your HOA or apartment complex started charging you a fee on top of your rent because you got more fedex packages than others in the neighborhood, would you be okay with that?  I sure wouldn't.

0 Likes
Re: What can we do to actually fix Verizon's Netflix throttling?
monsterlab
Enthusiast - Level 2

So, how do those of you who are still blaming Netflix explain this post from Level3?:

http://blog.level3.com/global-connectivity/verizons-accidental-mea-culpa/

Re: What can we do to actually fix Verizon's Netflix throttling?
nick_d
Enthusiast - Level 3

Haha Monstrlab, I was just coming to post that blog link myself... here are the contents for those who do not want to open a new tab (source: http://blog.level3.com/global-connectivity/verizons-accidental-mea-culpa/)


Verizon’s Accidental Mea Culpa

David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns Netflix’s customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by Verizon’s own paying broadband consumers.

His explanation for Netflix’s on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.

The diagram then shows this one little bar, suggestively color-coded in red so you know it’s bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post, I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.

But here’s the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as Verizon’s depiction of their own network. We engineer it that way. We have to maintain adequate headroom because that’s what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network – IN BOTH DIRECTIONS.

So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?

Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3’s network interconnects with Verizon’s in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.

So let’s look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of their content through this interconnection location. It is in a single building. And boils down to a router Level 3 owns, a router Verizon owns and four 10Gbps Ethernet ports on each router. A small cable runs between each of those ports to connect them together. This diagram is far simpler than the Verizon diagram and shows exactly where the congestion exists.

lvltvzw

Verizon has confirmed that everything between that router in their network and their subscribers is uncongested – in fact has plenty of capacity sitting there waiting to be used. Above, I confirmed exactly the same thing for the Level 3 network. So in fact, we could fix this congestion in about five minutes simply by connecting up more 10Gbps ports on those routers. Simple. Something we’ve been asking Verizon to do for many, many months, and something other providers regularly do in similar circumstances. But Verizon has refused. So Verizon, not Level 3 or Netflix, causes the congestion. Why is that? Maybe they can’t afford a new port card because they’ve run out – even though these cards are very cheap, just a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more. If that’s the case, we’ll buy one for them. Maybe they can’t afford the small piece of cable between our two ports. If that’s the case, we’ll provide it. Heck, we’ll even install it.

But, here’s the other interesting thing also shown in the Verizon diagram. This congestion only takes place between Verizon and network providers chosen by Netflix. The providers that Netflix does not use do not experience the same problem. Why is that? Could it be that Verizon does not want its customers to actually use the higher-speed services it sells to them? Could it be that Verizon wants to extract a pound of flesh from its competitors, using the monopoly it has over the only connection to its end-users to raise its competitors’ costs?

To summarize: All of the networks have ample capacity and congestion only occurs in a small number of locations, locations where networks interconnect with some last mile ISPs like Verizon. The cost of removing that congestion is absolutely trivial. It takes two parties to remove congestion at an interconnect point. I can confirm that Level 3 is not the party refusing to add that capacity. In fact, Level 3 has asked Verizon for a long time to add interconnection capacity and to deliver the traffic its customers are requesting from our customers, but Verizon refuses.

Why might that be? Maybe we should ask David Young.


0 Likes
Re: What can we do to actually fix Verizon's Netflix throttling?
Kestrel3
Contributor - Level 2

Could this problem also explain why many users have been seeing such poor actual upload speeds, but the speed tests to the closest Verizon servers look fine?  Uplolad speeds used to be great, but have dropped to about 10% of what they used to be.

see this thread:

http://forums.verizon.com/t5/FiOS-Internet/Slow-real-world-upload-speed/td-p/654849

0 Likes
Re: What can we do to actually fix Verizon's Netflix throttling?
PJL
Master - Level 3

@nick_d wrote:

@chomper87 wrote:
Verizon isn't chaning anything. Netflix has a catalog of videos. Netflix wants to stream those videos. Netflix needs to PAY for an ISP. Netflix needs to PAY for a CDN. What's so difficult to understand? If I start my own Video catalog and streaming service, should I get a free services from Verizon? Am I going to get a Free ISP? A free CDN? NO.

I'm not sure you understand how enterprise networks interconnect. Netflix IS ALREADY paying for their bandwidth from a top tier provider (at speeds greater than 1Tbps in many connection points and a fee of tens of thousands if not more). They ALREADY HAVE a CDN system using open peering. They're not trying to access the internet using bandwidth from Verizon like consumers do.  

Essentially you should think of Netflix as a really great burger place drawing tons of customers from all over. Those customers are using a highway to get to the burger place -- that highway is Verizon's network, Comcasts network, AT&Ts network, etc... a network for which the customers have already paid to access when they pay their monthy internet bill. So what verizon has done is what Chris Christie's office did in NJ -- they took their wide highway down to a lane or two and just let it back up. This is not Netflix demanding bandwidth from Verizon this is Verizon's customers demanding access to Netflix.  Netflix doesn't just send traffic without it being requested!! I don't know how that is so difficult to understand.  Verizon sees that a lot of it's customers are watching netflix and they feel like they should get more money for Netflix's success -- absolutely ridiculous.  It's like the mall seeing that a lot more people want to shop at a particular store they they try to charge that store more because they are popular - explain how that is fair.  If your HOA or apartment complex started charging you a fee on top of your rent because you got more fedex packages than others in the neighborhood, would you be okay with that?  I sure wouldn't.


I don't believe you take into account peering agreement contracts between the CDNs (aka content provider ISPs) and the consumer ISPs like Verizon.  That peering agreement contract may not be settlement-free.  I think it's a private peering agreement.

0 Likes
Re: What can we do to actually fix Verizon's Netflix throttling?
nick_d
Enthusiast - Level 3

@PJL wrote:

I don't believe you take into account peering agreement contracts between the CDNs (aka content provider ISPs) and the consumer ISPs like Verizon.  That peering agreement contract may not be settlement-free.  I think it's a private peering agreement.


They have a free open peering setup, verizon just doesn't like it very much: https://www.netflix.com/openconnect/guidelines

0 Likes
Re: What can we do to actually fix Verizon's Netflix throttling?
PJL
Master - Level 3

@nick_d wrote:

@PJL wrote:

I don't believe you take into account peering agreement contracts between the CDNs (aka content provider ISPs) and the consumer ISPs like Verizon.  That peering agreement contract may not be settlement-free.  I think it's a private peering agreement.


They have a free open peering setup, verizon just doesn't like it very much: https://www.netflix.com/openconnect/guidelines


Peering is between ISPs.  Netflix is not an ISP.  So Netflix Open Conect is not free peering between ISPs.  Apparently Verizon did not consider Open Connect to be advantageous to Verizon.  Most large consumer IPS did not; only the smaller ones did.

0 Likes
Re: What can we do to actually fix Verizon's Netflix throttling?
santosSoCal
Newbie

First of all, thank you for the abundant information you have provided.

Yes, I agree, the networking environment can be very complicated.

You seemed rather well versed on Verizon's position and service capabilities

but the issue is what kind of service is Verizon able to deliver to me.

I spoke with Version tech support and they verified that the speeds to/from my residence are

in compliance with the service level I pay for.

So, the conclusion the support technician and I reached was that, yes,  Netflix is being throttled.

The poor guy knows that this is a political/business hot potato, but couldn't offer any help.

Although you listed the variety of services that Verizon offers.  Do you happen to know which services

Netflix already subscribes to?

I, as a consumer, do not want to be pawn with the two heavyweights.

0 Likes
Re: What can we do to actually fix Verizon's Netflix throttling?
Hubrisnxs
Legend
http://publicpolicy.verizon.com/blog/entry/level-3s-selective-amnesia-on-peering


Apparently Level 3 is being intellectually dishonest with that latest post. They had the same complaint and were very public about it in the past. Now they have changed their story.
0 Likes