Re: DNS issues in SoCal
frnkblk
Enthusiast - Level 3

I stand corrrected, thanks.

0 Likes
Re: DNS issues in SoCal
laurastansfield

Hi All,

I too am having problems accessing my Multiple Listing Service via CRMLS Matrix at www.imrmls.com. I get to the Home Page but the login page to get into the MLS system won't come up and keeps giving me an Oops! Error Message. It says the webpage can't be found. I've called my technical support for CRMLS Matrix and they sent me an email with a link to this Forum and recommended that I contact Verizon FIOS technical support because hundreds of us are having problems with this. It is crippling my productivity and that of my colleagues quite badly and in our industry (Real Estate) that is the last thing we need right now! I am on the phone with Verizon FIOS technical support right now and they are going to collaborate with me to see if we can't get this resolved at least for the website I need to access. Will keep you all posted of my outcome.

0 Likes
Re: DNS issues in SoCal
houkou_onchi
Enthusiast - Level 3

Anyone still having problems with this? Someone posted similar issues on dslreports. I wonder if this has to do with how VZ does its weird load balancing depending on Odd/even IP.  If anyone is still having issues with this where the traceroute shows packets being dropped it might be interesting to try some other ips in the same /24 and see if it is specific to odd/even ips (You might notice that your 2nd (real) hop is a different router/IP depending on even/odd IP address. In my case this also causes a significant latency increase as well. Example:

traceroute to vnsc-bak.sys.gtei.net (4.2.2.2), 30 hops max, 46 byte packets
 1  router.houkouonchi.jp (1.1.1.1)  0.164 ms  0.133 ms  0.523 ms
 2  L103.LSANCA-DSL-29.verizon-gni.net (71.110.63.1)  1.026 ms  0.756 ms  0.724 ms
 3  G0-3-1-7.LSANCA-LCR-22.verizon-gni.net (130.81.146.60)  52.357 ms  39.216 ms  39.918 ms
 4  so-4-1-0-0.LAX01-BB-RTR2.verizon-gni.net (130.81.151.248)  127.499 ms  36.677 ms  1.965 ms
 5  0.ae2.BR3.LAX15.ALTER.NET (152.63.2.133)  2.655 ms  2.285 ms  2.105 ms
 6  4.68.63.245 (4.68.63.245)  3.378 ms  2.930 ms  3.413 ms
 7  ae-11-60.car1.LosAngeles1.Level3.net (4.69.144.3)  6.407 ms  2.594 ms *
 8  vnsc-bak.sys.gtei.net (4.2.2.2)  2.616 ms  2.485 ms  2.517 ms

root@dekabutsu: 07:50 AM :~# traceroute -I 4.2.2.3
traceroute to vnsc-lc.sys.gtei.net (4.2.2.3), 30 hops max, 46 byte packets
 1  router.houkouonchi.jp (1.1.1.1)  1.231 ms  0.133 ms  0.106 ms
 2  L103.LSANCA-DSL-29.verizon-gni.net (71.110.63.1)  1.419 ms  0.753 ms  0.728 ms
 3  G0-3-1-7.LSANCA-LCR-21.verizon-gni.net (130.81.140.224)  5.940 ms  5.479 ms  5.345 ms
 4  so-4-1-0-0.LAX01-BB-RTR1.verizon-gni.net (130.81.151.246)  5.749 ms  5.525 ms  5.656 ms
 5  0.ae1.BR3.LAX15.ALTER.NET (152.63.2.129)  5.867 ms  6.135 ms  5.869 ms
 6  4.68.63.245 (4.68.63.245)  6.303 ms  5.828 ms  5.841 ms
 7  ae-21-70.car1.LosAngeles1.Level3.net (4.69.144.67)  6.048 ms  6.154 ms *
 8  vnsc-lc.sys.gtei.net (4.2.2.3)  5.309 ms  5.796 ms  5.482 ms

root@dekabutsu: 07:50 AM :~# traceroute -I 4.2.2.4
traceroute to 4.2.2.4 (4.2.2.4), 30 hops max, 46 byte packets
 1  router.houkouonchi.jp (1.1.1.1)  0.237 ms  0.433 ms  0.140 ms
 2  L103.LSANCA-DSL-29.verizon-gni.net (71.110.63.1)  1.149 ms  1.274 ms  1.022 ms
 3  G0-3-1-7.LSANCA-LCR-22.verizon-gni.net (130.81.146.60)  1.891 ms  1.518 ms  1.252 ms
 4  so-6-0-0-0.LAX01-BB-RTR2.verizon-gni.net (130.81.29.126)  2.013 ms  1.923 ms  2.208 ms
 5  0.xe-11-1-0.BR1.LAX15.ALTER.NET (152.63.112.5)  2.270 ms  3.088 ms  2.369 ms
 6  ae6.edge1.LosAngeles9.level3.net (4.68.62.169)  3.105 ms  3.515 ms  3.042 ms
 7  ae-31-80.car1.LosAngeles1.Level3.net (4.69.144.131)  105.685 ms  228.988 ms  18.215 ms
 8  4.2.2.4 (4.2.2.4)  2.736 ms  2.515 ms  2.485 ms

root@dekabutsu: 07:50 AM :~# traceroute -I 4.2.2.5
traceroute to 4.2.2.5 (4.2.2.5), 30 hops max, 46 byte packets
 1  router.houkouonchi.jp (1.1.1.1)  0.247 ms  0.166 ms  0.145 ms
 2  L103.LSANCA-DSL-29.verizon-gni.net (71.110.63.1)  1.008 ms  0.814 ms  0.862 ms
 3  G0-3-1-7.LSANCA-LCR-21.verizon-gni.net (130.81.140.224)  5.608 ms  5.413 ms  5.338 ms
 4  so-4-1-0-0.LAX01-BB-RTR1.verizon-gni.net (130.81.151.246)  5.535 ms  5.540 ms  5.493 ms
 5  0.ae1.BR3.LAX15.ALTER.NET (152.63.2.129)  5.950 ms  5.933 ms  5.857 ms
 6  4.68.63.245 (4.68.63.245)  36.847 ms  8.010 ms  5.879 ms
 7  ae-11-60.car1.LosAngeles1.Level3.net (4.69.144.3)  5.346 ms * *
 8  4.2.2.5 (4.2.2.5)  5.478 ms  5.342 ms  5.478 ms


0 Likes
Re: DNS issues in SoCal
frnkblk
Enthusiast - Level 3

Even though this thread is labeled "DNS issues", there's been nothing shared that actually documents a DNS issue.  Everything has pointed to something in Verizon's network that's mishandling packets, packets on the return to the customer. 

@houkou_onchi: Yes, while your traceroutes show differnet paths depending on the destination IP address, that doesn't matter so much because they all complete.  So you haven't actually demonstrated an issue with DNS at all. 😃

If you can share a traceroute with us of sites that fail, that may be helpful, but the solution seems to be, in pretty well every case, getting another IP address that's not affected by whatever issue that Verizon has with their network gear.

0 Likes
Re: DNS issues in SoCal
PJL
Master - Level 3

I am seeing issues getting web responses only from certain sites (e.g. msnbc.com, barnsandnoble.com, abcnews.go.com, foxnews.com, etc.) when I use the Verizon's default DNS settings in Windows (obtain DNS server address automatically). (Other sites like dslreports.com and cnn.com are fine.) When I switch from the obtain DNS server address automatically setting (which gave me the 192.168.1.1 primary and 68.238.64.12) to the another DNS server -- Level 3 (4.2.2.2 and 4.2.2.3) - all sites are near-instantenous, including those that I could not get to before. Then I manually tried 68.238.64.12 (the Verizon server added for automatic settings) with no secondary and it failed -- and the Windows diagnostics said I had a DNS issue. I then tried 192.168.1.1 and got the same results. I then went back to the Level 3 DNS and all works perfectly. My router has been factory reset (twice).

My neighbor had the same issue.  Verizon released and renewed his IP address to give him a new WAN address -- and they broke his caller ID and remote DVR functions on FiOS TV!.  His Internet seems "better" though, but this band-aid fix introduces other problems.  I've put in a request on DSL Reports for this issue and I will try to get Verizon's attention Monday.


 

0 Likes
Re: DNS issues in SoCal
frnkblk
Enthusiast - Level 3

Can you  'dig' the fauly domain names using the bad DNS servers and share the results?  If the DNS servers truly don't work for you, we'll see bad/no results.

i.e.

dig msnbc.com @192.168.1.1

dig msnbc.com @68.238.64.12

0 Likes
Re: DNS issues in SoCal
PJL
Master - Level 3

I don't have dig, so I just used nslookup, which produced an interesting result.  When I used L3 4.2.2.2 DNS (which gives me great results browsing to msnbc.com) I got 65.55.251.214 for msnbc.com, a "problem site."  When I used 68.238.64.12 (the Verizon default DNS secondary) the result was 65.55.53.235 and when I used 192.168.1.1 (the Verizon default DNS primary) I also got 65.55.53.235 for msnbc.com.

So I tried barnsandnoble.com, also a "problem site."  With 4.2.2.2 I get 65.204.48.118  161.221.89.118.  With 68.238.64.12 I get  161.221.89.118  65.204.48.118 (backwards from the first), and the same for 192.168.1.1.

And for DSL report, a "non-problem site" With 4.2.2.2 I get 209.123.109.175.  With 68.238.64.12 I get 209.123.109.175 (the same), and the same for 192.168.1.1. 

But somehow I think this is still a routing issue?!?

Your thoughts please?

0 Likes
Re: DNS issues in SoCal
frnkblk
Enthusiast - Level 3

Thanks, PJL.  What this shows is that querying a different recursive DNS server returns a different IP address(es) .  This is

standard behavior for those websites that want to direct users to a different server (perhaps to a certain geographical region), which is done based on the "known" location of the recursive DNS server making the query.   I'm going to guess that's why, for some Verizon FiOS SoCal users and websites, changing the DNS server helps regain access.

What PJL's testing showed was that differnet IP addresses for the same website are accessible over Verizon's FiOS network in SoCal, which further confirms that something with the path (we believe return path) is affecting end-to-end connectivity.

On the outside chance that that routing is actually symmetric: PJL, can you provide traceroutes to 65.55.251.214 and 65.55.53.235, as well as 65.204.48.118 and 161.221.89.118?  Perhaps we can identify which path is the problem one.

0 Likes
Re: DNS issues in SoCal
houkou_onchi
Enthusiast - Level 3

@frnkblk: I myself never had this issue or it has been un-noticed. I just noticed routing changed for me a short while ago as I used to ping lower to odd ips now its even again. DNS has some weird policy based routing as they seem to do some of their load balancing just based off destination/source IP being even/odd/etc which is pretty weird to me as I have never seen any other network do that.

I was just showing an example with my traceroutes, of course they went through as I am not having issues. I was curious though if someone else who was having issues if it was just affecting odd or just even ips on the /24 they were trying to access.

0 Likes
Re: DNS issues in SoCal
frnkblk
Enthusiast - Level 3

@houkou_onchi wrote:

<snip>

I was curious though if someone else who was having issues if it was just affecting odd or just even ips on the /24 they were trying to access.


@houkou_onchi: Your theory about odd/even destination IP addresses doesn't seem to hold true, because your traceroutes documented in an earlier post to 4.2.2.2 and 4.2.2.5 both have second-last hops of ae-11-60.car1.LosAngeles1.Level3.net.

0 Likes