Re: Latency via Level3
infostruction
Newbie
0 Likes
Re: Latency via Level3
MMA1
Newbie

If this was resolved it is occurring again with a vengence.  It also looks like Verizon owns the company Alter.net which is the method items from the East Cost to the West Coast get routed.  Note that these issues occur at multiple verizon FiOS provided sites (tested at four different verizon FiOS supplied sites)- sample tracert's provided below. 

The verizon techs were nice enough but it's basically above their pay grade, their bosses pay grade and honestly probably can't be corrected until you go a few paygrades above all of those. 

What is my option - Comcast or simply get rid of the internet all together.  Most people would not even consider that...but the public who pays for services to be able to do activities whether it be telecommute, have their children complete homework or catch up on news, watch some shows or play some games....have to make a decision...pay or take your money and time and do something else.  Can you do it...well we don't have dish or cable - just an antenna. Do we have netflix...yes but we are considering canning that as well.  Am I complaining about my netflix streaming - no...it has nothing to do with netflix but we all know that there is a push and pull between both companies.  I am sure there is some credence to both sides of the arguement. Unfortunately - I am a customer and I don't care -figure it out because I do have options. 

business computer>tracert  asce.adobeconnect.com

Tracing route to na8cps.adobeconnect.com [216.104.214.109]
over a maximum of 30 hops:

  1     3 ms     5 ms     3 ms  Wireless_Broadband_Router.home [192.168.1.1]
  2    10 ms    11 ms    10 ms  L100.WASHDC-VFTTP-94.verizon-gni.net [96.241.211
.1]
  3    14 ms    14 ms    11 ms  G1-7-0-1.WASHDC-LCR-22.verizon-gni.net [100.41.1
95.82]
  4     *        *        *     Request timed out.
  5    19 ms    16 ms    18 ms  0.ae2.BR1.IAD8.ALTER.NET [140.222.229.165]
  6    14 ms    16 ms    15 ms  ae16.edge1.washingtondc12.level3.net [4.68.62.13
3]
  7    89 ms    91 ms    91 ms  vl-3603-ve-227.ebr2.Washington12.Level3.net [4.6
9.158.42]
  8    88 ms    91 ms    87 ms  ae-57-57.ebr1.NewYork1.Level3.net [4.69.201.65]

  9    90 ms    92 ms    93 ms  ae-61-61.csw1.NewYork1.Level3.net [4.69.134.66]

 10    92 ms    87 ms    93 ms  ae-62-62.ebr2.NewYork1.Level3.net [4.69.148.33]

 11    89 ms    90 ms    91 ms  ae-2-2.ebr1.SanJose1.Level3.net [4.69.135.185]
 12    94 ms    98 ms    98 ms  ae-61-61.csw1.SanJose1.Level3.net [4.69.153.2]
 13    91 ms    91 ms    92 ms  ae-1-60.edge1.SanJose3.Level3.net [4.69.152.16]

 14    93 ms    91 ms    92 ms  ADOBE-SYSTE.edge1.SanJose3.Level3.net [4.53.208.
98]
 15    92 ms    90 ms    90 ms  na8cps.adobeconnect.com [216.104.214.109]

Trace complete.

Tracing route to cnn.com [157.166.226.26]

over a maximum of 30 hops:

1   <1 ms   <1 ms   <1 ms Wireless_Broadband_Router.home [192.168.1.1]

2     8 ms     9 ms     9 ms L100.WASHDC-VFTTP-94.verizon-gni.net [96.241.2

.1]

3   12 ms     9 ms   10 ms G0-13-3-4.WASHDC-LCR-21.verizon-gni.net [130.8

109.170]

4     *       *       *     Request timed out.

5   13 ms   15 ms   15 ms 0.ae1.BR2.IAD8.ALTER.NET [140.222.229.167]

6   12 ms   15 ms   10 ms ae17.edge1.washingtondc12.level3.net [4.68.62.

7]

7   24 ms   21 ms   20 ms vl-3504-ve-118.ebr1.Washington12.Level3.net [4

9.158.30]

8   20 ms   21 ms   20 ms ae-6-6.ebr1.Atlanta2.Level3.net [4.69.148.105]

9   24 ms   21 ms   21 ms ae-1-51.edge4.Atlanta2.Level3.net [4.69.150.13

10     *       *       *     Request timed out.

11     *       *       *     Request timed out.

12     *       *       *     Request timed out.

13     *       *       *     Request timed out.

14     *       *       *     Request timed out.

15     *       *       *     Request timed out.

16     *       *       *     Request timed out.

17     *       *       *     Request timed out.

18     *       *       *     Request timed out.

19     *       *       *     Request timed out.

20     *       *       *     Request timed out.

21     *       *       *     Request timed out.

22     *       *       *     Request timed out.

23     *       *       *     Request timed out.

24     *       *       *     Request timed out.

25     *       *       *     Request timed out.

26     *       *       *     Request timed out.

27     *       *       *     Request timed out.

28     *       *       *     Request timed out.

29     *       *       *     Request timed out.

30     *       *       *     Request timed out.

Trace complete.

Personal computer>tracert aol.com

Tracing route to aol.com [205.188.101.58]

over a maximum of 30 hops:

1   <1 ms   <1 ms   <1 ms Wireless_Broadband_Router.home [192.168.1.1]

2     9 ms     8 ms     9 ms L100.WASHDC-VFTTP-94.verizon-gni.net [96.241.21

.1]

3   11 ms   13 ms     9 ms G1-7-0-0.WASHDC-LCR-21.verizon-gni.net [100.41.

95.76]

4     *       *       *     Request timed out.

5   10 ms     9 ms   10 ms 0.ae1.BR2.IAD8.ALTER.NET [140.222.229.167]

6   14 ms     9 ms   10 ms pop1-ash-xe-1-2-0.atdn.net [66.185.139.225]

7   15 ms   15 ms   10 ms dar1-dtc-xe-1-1-0.atdn.net [66.185.151.158]

8   11 ms   15 ms   15 ms gear1-dtc-10GigabitEthernet1-1.net.aol.com [66.

85.150.34]

9   12 ms   15 ms   15 ms edge3-dtc-ae0.net.aol.com [149.174.32.10]

10     *       *       *     Request timed out.

11     *       *       *     Request timed out.

12     *      *       *     Request timed out.

13     *       *       *     Request timed out.

14     *       *       *     Request timed out.

15     *       *       *     Request timed out.

16     *       *       *     Request timed out.

17     *       *       *     Request timed out.

18     *       *       *     Request timed out.

19     *       *       *     Request timed out.

20     *       *       *     Request timed out.

21     *       *       *     Request timed out.

22     *       *       *     Request timed out.

23     *       *       *     Request timed out.

24     *       *       *     Request timed out.

25     *       *       *     Request timed out.

26     *       *       *     Request timed out.

27     *       *       *     Request timed out.

28     *       *       *     Request timed out.

29     *       *       *     Request timed out.

30     *       *       *     Request timed out.

Trace complete.

0 Likes
Re: Latency via Level3
Hubrisnxs
Legend

You guys are looking at your traceroutes incorrectly.

Great guide for how to diagnose network issues using traceroutes.

https://www.linode.com/docs/networking/diagnosing-network-issues-with-mtr#analyzing-mtr-reports

These traces that you are showing, show the trouble clearly on level3's end and well beyond the verizon network in some cases.

0 Likes
Re: Latency via Level3
jkiechlin
Newbie

I am a network admin and I have tried to support the end-user above (MMA) who is a teleworker trying to use Adobe Connect. I know she may have misinterpreted the traceroute, but there is an issue between Level3 and Verizon. I think any end-user who is told by Verizon the problem is on the Level3 network has no recourse. Level3 is the backbone (sorry Verizon) of the Internet. Verizon should own the issue and communicate with Level3 on the subscribers behalf. And this problem is not limited to a single user. Since the assitance she asked for is work related, you would hope that Verizon techs woudln't dismiss the issue so easily. Telecommuters can lose their jobs over these kinds of issues.

This user tested with multiple neighbors who had FIOS in Manssas, VA. All failed miserably. Those who had Comcast (and slower connections) have no troubles. So it is not the laptop (which we replaced).

I tested from my home (FIOS location) in Leesburg, VA. Same problems. It shows the same latency at the second hop on Level3 (72-90).

All traceroutes with Verizon-DC show Verizon-> Alternet -> Level3 ->Adobe Connect in Palto Alto, CA

Now here's the strange part. When I test from other ISPs, all traceroutes show latency (72-121 ms) on the Level3 hops, but the service works flawlessly. No lag and always passes the adobe network test. Hmmm, it almost defies logic.

I can use my AT&T MiFI with 20MB down and 5MB up and it outperforms my home FIOS connection at 35/35 for the Adobe Connect (and other Level3 sites as well).

Here's my AT&T MiFi trace with worse latency on the traceroute than the user showed. Yet it works, and very fast!

Tracing route to na8cps.adobeconnect.com [216.104.214.109]
over a maximum of 30 hops:

  1     4 ms     2 ms     2 ms  att.mifiliberate [192.168.1.1]
  2    51 ms    42 ms    35 ms  172.26.242.123
  3    70 ms    47 ms    63 ms  172.26.236.2
  4    51 ms    36 ms    39 ms  172.26.96.2
  5    50 ms    40 ms    59 ms  172.26.96.193
  6    57 ms    80 ms    41 ms  172.16.118.241
  7    52 ms    58 ms    60 ms  12.249.2.9
  8    74 ms    69 ms    58 ms  12.83.188.242
  9    77 ms    65 ms    60 ms  cr1.cgcil.ip.att.net [12.122.2.205]
 10    74 ms    60 ms    57 ms  ggr4.cgcil.ip.att.net [12.122.133.33]
 11   102 ms    70 ms    58 ms  192.205.37.150
 12   112 ms   111 ms   116 ms  vl-3506-ve-120.ebr1.Chicago2.Level3.net [4.69.158.134]
 13   125 ms   107 ms   108 ms  ae-3-3.ebr2.Denver1.Level3.net [4.69.132.61]
 14   136 ms   103 ms   113 ms  ae-1-100.ebr1.Denver1.Level3.net [4.69.151.181]
 15   134 ms   114 ms   108 ms  ae-3-3.ebr2.SanJose1.Level3.net [4.69.132.57]
 16   114 ms   118 ms    97 ms  ae-72-72.csw2.SanJose1.Level3.net [4.69.153.22]
 17   112 ms   100 ms    97 ms  ae-2-70.edge1.SanJose3.Level3.net [4.69.152.80]
 18   113 ms    99 ms    99 ms  ADOBE-SYSTE.edge1.SanJose3.Level3.net [4.53.208.98]
 19   150 ms   121 ms   108 ms  na8cps.adobeconnect.com [216.104.214.109]

Here's the test URL for Adobe Connect:

https://na1cps.adobeconnect.com/common/help/en/support/meeting_test.htm

I have polled all of our employees who have FIOS in the DC area, and they all have issues connecting to AC in Palo Alto, CA.

I am starting to think there are some contractual and peering issues between Level3 and Verizon. Level3 has a large number of CDN customers: Netflix, FOX, Yahoo!, Microsoft, Comcast, Sky, NHL, BBC, DNC, CNN and others. Several of the level3 sites are having the same poor performance.

http://www.dslreports.com/shownews/Level3-Verizon-Intentionally-Causing-Netflix-Congestion-129745

Can anyone who is a FIOS user in the DC area try the URL:

https://na1cps.adobeconnect.com/common/help/en/support/meeting_test.htm

and post whether they were succesful. I am showing a "modem" connection speed in Leesburg, VA, which is bad. If it's good it will say "Lan Speed".After the test if you click the details screen, it shows the latency and up/down speeds. A tracenet would be nice if you are so inclined.

Of course I'm not sure where I can take the info. I am working on the adobe end also. But it might prove that many Verizon customers in the DC area are suffering poor service across Level3, and Comcast customers are not.

0 Likes
Re: Latency via Level3
Hubrisnxs
Legend

Unfortunately when communicating on the forums, you are talking mostly to other FiOS Users.  Verizon employee's moderate the forums, in a true moderator fashion and occasionally take billing and account related issues offline. 

Here is what verizon posts at the top of the forums


The Verizon Community Forums is where customers help other customers.
If you need assistance for specific Verizon account issues, please contact us. .


Alternate methods that have proven succesful are the social media support groups at Twitter  or the Verizon Direct forums at Broadband Reports.

might want to give them a try.

0 Likes
Re: Latency via Level3
MMA1
Newbie

Respectfully, you should note that I have already had technicians to my location.  The technicians are seeing the same thing I see.  I do not experience the same latency issues (regardless of whether I truly understand exactly where the issue begins and explain it properly as I am truly an end user doesn't matter).  If I try to access any site located lets say on the East Coast my tracerts and the "internet roadway" I am sent over is different and does not display the same results/issues.  I do not have issues with items going to different places except when I am trying to reach something on the West Coast and then I am routed along the same roadway through the same locations and have problems.  We have now idenitfied two other individuals living in VA with Verizon Fios who are experiencing similar problems and showing similar latency and tracerts.

I posted here because I kept getting search results that pointed to this forum and I wanted to know if other people who had posted previously or new individuals were seeing similar issues.

The Verizon techs have closed the tickets...they tried to escalate the issue.  Verizon has basically washed their hands of the issue.  I am still currently their customer.  So I AM trying to talk to other usersIn the most polite way as possible I am saying Verizon doesn't care and has basically taken that stance.  I am looking for assitance and information.  If you can't provide any other than go talk to Verizon, please don't reply.  It doesn't help me.

Now I get to decide what I want to do and I am trying to see what others are experiencing.  I thank you for your reply and acknowledge I am limited in my understanding - however I have had multiple technical experts in Networking tell me to contact Verizon as it is something they must and should pursue as I am their customer.  These same techs admit that it will probably never get addressed as I am a very small fish in a big sea and Verizon as well  as other companies like them simply don't care.  So does Verizon really have the best customer service like they used ....I am beginning to formulate an opinion on this and I don't think Verizon would like it.

0 Likes
Re: Latency via Level3
Hubrisnxs
Legend

IMHO as a networking professional with decades of experience, your only solution is to get A VPN.

verizon only controls there portion of the network and can only make suggestions that level 3 or Any cTheseompany increase there bandwidth. The other company can not be forced to do anything by Verizon.

A) the data shown shows no trouble in vz's network.
B) the traces shown are incomplete.

Tracert is a command that measures round trip latency but only displays the forward trip and doesn't display the reverse path. You NEED the second trace from the company server back to you in order to truly find out where the problem is. The reverse trace will show what path and choke points are really affecting your service.

If you want the long answer then you'll have to do a lot more homework unfortunately. Homework that the professionals helping you failed to do with,for you.

If you want the short answer, then you'll want to get A VPN.

Additionally these people that are assisting you should be able to help you Spott The Fake Latency

The Rule is If there is an actual issue, the latency or packet drops will continue or increase for all future hops:

Latency spikes and ***'s in the middle of a traceroute mean nothing if they do not continue forward.

• At worst it could be the result of an asymmetric path.
• But it is probably an artificial rate-limit or prioritization issue.
• By definition, if regularly forwarded packets are being affected
you should see the issue persist on all future hops.

It would behoove you to go through the link I shared, or share it with your Support staff that is helping you.  You'll find as pointed out in this power point presentation that


Most modern networks are actually well run

So simple issues like congestion or routing loops are becoming

a smaller percentage of the total network issues encountered.

And more commonly, the encountered issues are complex

enough that a naïve traceroute interpretation is utterly useless.

Few people are skilled at interpreting traceroute

Most ISP NOCs and even most mid-level engineering staff are

not able to correctly interpret complex traceroutes.

This leads to a significant number of misdiagnosed issues and

“false reports”, which flood the NOCs of networks worldwide.

In many cases the problem of false reports is so bad, it is all

but impossible for a knowledgeable outside party to submit a

traceroute related ticket about a real issue


 

0 Likes
Re: Latency via Level3
Hubrisnxs
Legend
Fyi. The reason I pointed you in the direction of those two arms of verizon is because from my experience, they would be the most adept at diagnosing the problem and providing you with a possible solution. Field techs are not networking Guru's. They rely on actual network and tier ii engineers for these types of problems. I was giving you a short cut to those people.
0 Likes
Re: Latency via Level3
jlafferty
Newbie

Now its alternet thats borked.

  1     3 ms     3 ms     3 ms  Wireless_Broadband_Router.home [192.168.1.1]
  2     5 ms     5 ms     4 ms  xxxxxxxxx
  3     9 ms    11 ms    10 ms  G1-7-1-1.WASHDC-LCR-22.verizon-gni.net [100.41.2
03.80]
  4    23 ms     5 ms     6 ms  ae6-0.RES-BB-RTR2.verizon-gni.net [130.81.199.14
6]
  5     *        *        *     Request timed out.
  6     *        *        *     Request timed out.
  7    10 ms     7 ms     7 ms  0.ae4.XT2.DCA6.ALTER.NET [140.222.225.131]
  8    10 ms     8 ms     7 ms  80.ae1.SA4.DCA6.ALTER.NET [152.63.41.114]
  9     *        *        *     Request timed out.
 10     *        *        *     Request timed out.
 11     *        *        *     Request timed out.
 12     *        *        *     Request timed out.
 13     *        *        *     Request timed out.
 14     *        *        *     Request timed out.
 15     *        *        *     Request timed out.
 16     *        *        *     Request timed out.
 17     *        *        *     Request timed out.
 18     *        *        *     Request timed out.
 19     *        *        *     Request timed out.
 20     *        *        *     Request timed out.
 21     *        *        *     Request timed out.
 22     *        *        *     Request timed out.
 23     *        *        *     Request timed out.

0 Likes
Re: Latency via Level3
MMA1
Newbie

Yes and it looks Verizon knows alter.net....but lets see if I am interpreting the information correctly....

WHOIS information for alter.net:**

[Querying whois.verisign-grs.com]
[Redirected to whois.markmonitor.com]
[Querying whois.markmonitor.com]
[whois.markmonitor.com]
Domain Name: alter.net
Registry Domain ID: 
Registrar WHOIS Server: whois.markmonitor.com
Registrar URL: http://www.markmonitor.com
Updated Date: 2014-07-18T11:24:33-0700
Creation Date: 1990-07-25T21:00:00-0700
Registrar Registration Expiration Date: 2015-07-24T21:00:00-0700
Registrar: MarkMonitor, Inc.
Registrar IANA ID: 292
Registrar Abuse Contact Email: abusecomplaints@markmonitor.com
Registrar Abuse Contact Phone: +1.2083895740
Domain Status: clientUpdateProhibited
Domain Status: clientTransferProhibited
Domain Status: clientDeleteProhibited
Registry Registrant ID: 
Registrant Name: Verizon Business Global LLC
Registrant Organization: Verizon Business Global LLC
Registrant Street: One Verizon Way
Registrant City: Basking Ridge
Registrant State/Province: NJ
Registrant Postal Code: 07920
Registrant Country: US
Registrant Phone: +1.7033513164
Registrant Phone Ext: 
Registrant Fax: +1.7033513669
Registrant Fax Ext: 
Registrant Email: domainlegalcontact@verizon.com
Registry Admin ID: 
Admin Name: Domain Administrator
Admin Organization: Verizon Trademark Services LLC
Admin Street: 1320 North Court House Road
Admin City: Arlington
Admin State/Province: VA
Admin Postal Code: 22201
Admin Country: US
Admin Phone: +1.7033513164
Admin Phone Ext: 
Admin Fax: +1.7033513669
Admin Fax Ext: 
Admin Email: domainlegalcontact@verizon.com
Registry Tech ID: 
Tech Name: Verizon Business Technical Support
Tech Organization: Verizon
Tech Street: 1320 North Court House Road
Tech City: Arlington
Tech State/Province: VA
Tech Postal Code: 22201
Tech Country: US
Tech Phone: +1.8009000241
Tech Phone Ext: 
Tech Fax: -
Tech Fax Ext: 
Tech Email: help4u@verizonbusiness.com
Name Server: auth200.ns.uu.net
Name Server: auth00.ns.uu.net
Name Server: auth60.ns.uu.net
Name Server: auth210.ns.uu.net
URL of the ICANN WHOIS Data Problem Reporting System: http://wdprs.internic.net/
<<
The Data in MarkMonitor.com's WHOIS database is provided by MarkMonitor.com for
information purposes, and to assist persons in obtaining information about or
related to a domain name registration record.  MarkMonitor.com does not guarantee
its accuracy.  By submitting a WHOIS query, you agree that you will use this Data
only for lawful purposes and that, under no circumstances will you use this Data to:
 (1) allow, enable, or otherwise support the transmission of mass unsolicited,
     commercial advertising or solicitations via e-mail (spam); or
 (2) enable high volume, automated, electronic processes that apply to
     MarkMonitor.com (or its systems).
MarkMonitor.com reserves the right to modify these terms at any time.
By submitting this query, you agree to abide by this policy.
MarkMonitor is the Global Leader in Online Brand Protection.
MarkMonitor Domain Management(TM)
MarkMonitor Brand Protection(TM)
MarkMonitor AntiPiracy(TM)
MarkMonitor AntiFraud(TM)
Professional and Managed Services
Visit MarkMonitor at http://www.markmonitor.com
Contact us at +1.8007459229
In Europe, at +44.02032062220
0 Likes