Re: DNS issues in SoCal
Porch1
Newbie

I have clients all over the area that can't get to some sites. craigslist.org is often not working. Some can't get to Epson's FTP site for driver downloads. And I can't get to one of my email provider.  Most are on DSL, but some are on Fios. 

All work fine on my T-Mobile phone. 

0 Likes
Re: DNS issues in SoCal
mrballcb
Enthusiast - Level 3

@JamesLaszko wrote:

To add insult to injury, I can now not access forums.verizon.com through my FIOS connection either:


Page times out

Oh, wait, it must be the fault of the web site that we're going to.


I was troubleshooting last week why I could not get to a fastly.net CDN node (the west coast node, the Dallas and Virginia nodes worked just fine if I hardcoded those IP's into my /etc/hosts file).  I was in IRC with the fastly.net admins and they could see my packets getting to the west coast node (both icmp and tcp), and they could see the return packets getting sent back to me, but those packets never got back to me.  It was the same issue in that I could not reach the
One opinion I got was that Verizon needs to go and start resetting some line cards, it sounds like they have some corrupted src+dst flow tables somewhere.  Another is a replacement box with a misconfiguration that's blocking certain packets.  Either way, it's been going on since about Mon Jan 8.
The only solution (not a solution at all, just a hack around the problem) is rebooting your fios gateway and trying to get a different IP.  You will likely need to call tech support and get them to invalidate your lease to ensure you get a new IP.
...Todd
0 Likes
Re: DNS issues in SoCal
frnkblk
Enthusiast - Level 3

Perhaps the change in IP address results in a different hash, and so the return traffic goes over a different port in the LAG (or whatever load-balancing/port aggregation scheme they have in place.)

Re: DNS issues in SoCal
mrballcb
Enthusiast - Level 3

@frnkblk wrote:

Perhaps the change in IP address results in a different hash, and so the return traffic goes over a different port in the LAG (or whatever load-balancing/port aggregation scheme they have in place.)


Exactly what we were thinking.

Re: DNS issues in SoCal
JamesLaszko
Enthusiast - Level 3

The hard part of things is that VZ uses an internal MPLS network to distribute traffic to the "best" place for off-loading.  Even geographically close sites may end up routing through completely geographically distributed sites, based on least cost or best route available routing.  I'm at a loss that VZ NOC staff are not able to monitor issues like we have been seeing over the last week.....

We've been receiving a lot of support calls for one of our customers, Altura Credit Union.  They've had a ton of customers calling with connectivity issues - go figure, they are in the Inland Empire.  If there are any Altura customers watching this thread - Altura has been beating the crap out of VZ and us to make sure these issues go away - problem is that it's soley under the control of the VZ people.

Big pain....

James

Re: DNS issues in SoCal
frnkblk
Enthusiast - Level 3

I'd hope that Verizon is working night and day on this issue, but based on the lack of communication and success, I'm skeptical.

0 Likes
Re: DNS issues in SoCal
loy2
Newbie

after reading the messages, i decided to use chat live to check the status (i couldn't connect to steampowered and my wife couldn't connect to aol unless she goes through help.aol.com).  anyway, miguel, the tech guy, had me reset the router (i did this at least 3 times for the past week and a half) and after rebooting, i could now connect to these sites...so it might be worthwhile for those still having problems to use live chat...it worked for me...hopefully, this works for you guys too.

PS.  he mentioned something about a a break on the dhcp...but it solved my problem.....

0 Likes
Re: DNS issues in SoCal
mrballcb
Enthusiast - Level 3

@loy wrote:

after reading the messages, i decided to use chat live to check the status (i couldn't connect to steampowered and my wife couldn't connect to aol unless she goes through help.aol.com).  anyway, miguel, the tech guy, had me reset the router (i did this at least 3 times for the past week and a half) and after rebooting, i could now connect to these sites...so it might be worthwhile for those still having problems to use live chat...it worked for me...hopefully, this works for you guys too.

PS.  he mentioned something about a a break on the dhcp...but it solved my problem.....


Yes, he manually "broke your lease", which is a way of saying he made your current dhcp IP assignment invalid so that when you rebooted your router, it ensured you got a new IP address.  We've been saying all along that it's a function of your IP assignment as to whether or not this issue appears.  But they're not fixing the problem when they have you do this.

This clearly indicates that, depending on what IP you have, if you have one of the bad ones your traffic gets routed across some distinct piece of equipment or combination of equipment or through some peer that has an incomplete path or is misconfigured.  Again, they're not fixing the problem when they give you a new IP.  The problem still exists within Verizon's infrastructure, but you're getting an IP that gets handled/routed differently and your new route doesn't experience the problem.  It's like your city's public works department telling you that the way to fix the potholes on your drive to work down Main Street is to take 2nd Street to get to work.

VZ's noc needs to be able to replicate it and have someone in real time be communicating with them so they can trace the packets as it traverses their network, until they find the point that it's being dropped (and I strongly suspect it will be the return path dropping, not the forward path).  My testing with the admins of the CDN node I was unable to reach confirmed that, at least in my case.

I'm in Murrieta, BTW.

Re: DNS issues in SoCal
mrballcb
Enthusiast - Level 3

Oh, and I forgot to mention: in the original issue I had, which was last week, one of the sites I was unable to reach was forums.verizon.net, identical to one other poster who said he couldn't get to it at all.

0 Likes
Re: DNS issues in SoCal
RileyCat
Newbie

This has been going on in Temecula for over a week now. 

I have an open trouble ticket and can't get the issue solved.  I spoke with Rick at tech support again this AM and all that he could say was that it was an open issue and that it had been escallated.  He assured me that it would be resolved soon, yet when I asked what the issue was and how they planned to fix it he couldn't offer any information.

The orignaltechnicians I spoke with were clueless.  They asked me to clear the cache on all my browers and said that would clear it up. I didn't think it was funny. 

I can access craigslist.org from the starbucks down the street or from my work provided MiFi device and phone (Verizon) but not from any of my computers via Fios at home.

I can do a trace route to Craigslist..


C:\Users\Scott>tracert www.craigslist.org

Tracing route to www.craigslist.org [208.82.238.130]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  my.router [192.168.2.1]
  2     1 ms    <1 ms    <1 ms  myrouter.home [192.168.1.1]
  3     7 ms     7 ms     7 ms  L100.LSANCA-VFTTP-121.verizon-gni.net [173.58.73
.1]
  4    13 ms    12 ms    12 ms  G10-0-4-1921.LSANCA-LCR-09.verizon-gni.net [130.
81.138.84]
  5    15 ms    11 ms    13 ms  so-4-0-0-0.LAX01-BB-RTR1.verizon-gni.net [130.81
.28.72]
  6    16 ms    12 ms    12 ms  0.ae1.BR3.LAX15.ALTER.NET [152.63.2.129]
  7    12 ms    12 ms    12 ms  lap-brdr-03.inet.qwest.net [63.146.26.209]
  8    23 ms    21 ms    23 ms  svl-edge-23.inet.qwest.net [67.14.12.226]
  9    21 ms    22 ms    22 ms  65.113.32.14
 10    23 ms    21 ms    22 ms  vs-imgs-f.craigslist.org [208.82.238.130]

Trace complete.

Why can't Verizon resolve this?

0 Likes