Verizon (or someone) dropping packets from Uni of Pitt
Kepler2
Enthusiast - Level 2

I work for Pitt, and we have a Verizon FiOS residential customer who is unable to access any websites in a particular subnet. We actually had the same issue with someone who is geographically in the same area, and gets an IP from the same range (108.32.0.0/12) a few months ago. That person actually ended up switching to Comcast to fix the problem.

I personally have FiOS and get an IP in the 74.96.0.0/12 range and have no issues, as does several other employees that we had test...just none in the 108.32.0.0/12 range.

I had the user run a packet capture, and I ran one on our load balancer, and we can see her packets leaving correctly, hitting the load balancer, and the load balancer doing its job, and returning packets, but the user never gets any return packets. All we see from their side is retransmissions.

We have a ticket opened with Verizon, but I don't want to run into the same issue where the user gets fed up with no one being able to help on Verizon's side that they leave.

I'm not sure if any network engineers from Verizon read this board, or if problems can be escalated. It is a problem when a student is forced to use their cell phone tethering to access their school's website.

Thanks!

0 Likes
1 Solution

Correct answers
Re: Verizon (or someone) dropping packets from Uni of Pitt
Kepler2
Enthusiast - Level 2

It was the DSCP tagging. I set up a web server in our development environment and added DSCP tagging and the user was not able to pull a website, despite me seeing traffic from her address coming in via tcpdump, and responding correctly.

Once I removed the DSCP tagging from the zone, the user was able to get to the website without issue. So I tested it twice more -- adding/removing and it is the problem.

I then performed traceroutes to the user, with and without DSCP tagging, and it always follows the same path, ending on a Verizon node, so that node, or another after it that isn't responding to my ICMP is either re-routing, or dropping traffic tagged with DSCP 43. That is 100% a misconfiguration on Verizon's part, as they should not trust DSCP tags coming from outside networks.

View solution in original post

Re: Verizon (or someone) dropping packets from Uni of Pitt
Hubrisnxs
Legend

Hi there.  They don't normally, - the site is a peer to peer forum, with some standard forum administraton, but we've seen a lot of users get out of a range like that fairly easily by following the steps outlined below.

Pls feel free to report back and let us know if it helped.

 They have to log into the Verizon router.  192.168.1.1    user name is admin and the password is printed on the sticker as "password" or "login password"   People like to put the wpa2 or WEP key here for some reason, so try to avoid that, this is different then that.  Use these instructions to log in

How to find Router Default Username and Password | FiOS ...

 Then follow these steps

  1. Actiontec MI424-WR - RELEASE DHCP
    Click on MY NETWORK icon at the top.
    Select NETWORK CONNECTIONS from the menu on the left.
    Select BROADBAND CONNECTION (coax or ethernet) depending on your connection to the ONT.
    Click SETTINGS
    Click RELEASE
    Click APPLY
    Disconnect the router immediately to prevent it from re-requesting a DHCP lease.

After you have disconnected it, if you reconnect it too fast, it will simply get the same IP and or the same subnet. So you want to leave it disconnected for at least 5 full minutes, and preferably 10 -15.   After that wait period, reconnect it, check the IP again and it should be at the least a different subnet.

Re: Verizon (or someone) dropping packets from Uni of Pitt
Kepler2
Enthusiast - Level 2

Thanks for the look!

The user was sent a new router by Verizon prior to calling us -- So the different MAC requesting an IP should have accomplished the same thing. Would it be worth having the user unplug the cable/battery from the box installed in their basement, where the fiber terminates? I'm iffy on recommending this since they have to unscrew that bolt that says that only technicians should do that.

I should mention that she went through three calls with Verizon, so they did the power cycling, hardware replacement, and normal troubleshooting. The bigger issue is that I remembered that a few months ago, someone had the same problem from the same IP block.

0 Likes
Re: Verizon (or someone) dropping packets from Uni of Pitt
Hubrisnxs
Legend

@Kepler wrote:

Thanks for the look!

The user was sent a new router by Verizon prior to calling us -- So the different MAC requesting an IP should have accomplished the same thing. Would it be worth having the user unplug the cable/battery from the box installed in their basement, where the fiber terminates? I'm iffy on recommending this since they have to unscrew that bolt that says that only technicians should do that.

I should mention that she went through three calls with Verizon, so they did the power cycling, hardware replacement, and normal troubleshooting. The bigger issue is that I remembered that a few months ago, someone had the same problem from the same IP block.



The normal troubleshooting including getting a new router, won't accomplish any of what you need.  The process outlined will in fact get you a new ip on different subnets.

The router, and you can test this yourself, when you unplug it and plug it back in, you in fact get literally the same IP address back.

That defeats the whole purpose.

The release of the IP and Mac Binding is Key, but the more important thing is the length of time you leave it disconnected.

In fact even if you had a Verizon rep on the line and asked him to clear that arp/mac, and you rebooted it with a Verizon tech on the phone, you would still get the same ip.

The ONT Doesn't cary the ARP table, so rebooting it won't be of any use to you.  

0 Likes
Re: Verizon (or someone) dropping packets from Uni of Pitt
Kepler2
Enthusiast - Level 2

It would accomplish what I need -- you said it yourself:

"The release of the IP and Mac Binding is Key, but the more important thing is the length of time you leave it disconnected".

Getting a whole new router, where the WAN port has a whole new MAC address makes the DHCP lease for the old MAC irrelevant. 

"The process outlined will in fact get you a new ip on different subnets."

I don't see how this gets a new IP on a different subnet -- if the DHCP request from the WAN interface to ff:ff:ff:ff:ff:ff is being responded to by whatever DHCP server is serving the 108.32.x.x addr, why would that server decide to pick a new range? Unless there was a configuration issue that allows DHCP requests to that 108.32.x.x DHCP server (bad DHCP relay address on the layer 3 maybe), which has been fixed, this may not work.

I know clearing ARP wouldn't fix it, as the DHCP server doesn't care about the router's ARP table.

What you described was trying to get the DHCP lease gone -- which a new MAC address on a new modem would do.

0 Likes
Re: Verizon (or someone) dropping packets from Uni of Pitt
Kepler2
Enthusiast - Level 2

Also note: All other web sites work -- just not a certain subnet from Pitt. The only thing special about this subnet is that we tag all packets with a DSCP value of 43 (basically any traffic that goes to 136.142.34.0/23 forgoes our firewall and is routed directly to a load balancer), and that value of 43 is what keeps that traffic flowing correctly.

When I do a capture at home, that value is not stripped anywhere along the way. My guess is that Verizon, or one of the peering links between us and wherever that subnet is served from is dropping traffic with DSCP tags, or they are doing the same thing we are -- changing the route based on DSCP tags.

0 Likes
Re: Verizon (or someone) dropping packets from Uni of Pitt
Hubrisnxs
Legend
Most people, yourself included, don't don't a router and then wait 5-15 minutes to connect a new one. So I'm not quite sure what you mean by that, but verizon areas are served by several gateway routers handing out ip's to its customers from various subnets. The proof is simply in the pudding the method mentioned works flawlessly.
0 Likes
Re: Verizon (or someone) dropping packets from Uni of Pitt
Hubrisnxs
Legend
Also a new Mac on a new modem would cause the connection to flat out stop working. Verizon does very strict Mac authentication and binding. The second a new Mac attempts to connect, the gateway refuses the connection. Calling in, they clear that binding and a new home router gets connected in seconds (again defeating the purpose) Rather than the time described. The time described only works if the lease is broken and the connection is sitting vacant and not occupying an address block. This process allows enough time for the ip to go back into the pool of ips and get cycled to a new customer. These routers are employed in a tier I network and serving an impressively large customer base. That's how or why it's typically going to throw you on a different subnet vs the same, largely because that subnet is likely close to a threshold at various parts of the day.
0 Likes
Re: Verizon (or someone) dropping packets from Uni of Pitt
Kepler2
Enthusiast - Level 2
I was able to contact the user, the steps in post one did not work. User obtained an IP in the same range. She left it disconnected for nearly 2.5 hours while she went to class.
0 Likes
Re: Verizon (or someone) dropping packets from Uni of Pitt
Hubrisnxs
Legend

2 and a half hours should have worked, the Verizon lease on the IP address is at 2 hours.  Some area's might be set to their old 4 hour lease, BUT if you did the release dhcp and immediately turned it off (if it wasn't it would have grabbed that ip within seconds) then that might have been an issue. 

I can tell you that 90% of people here don't need to leave it off for any more than 5 minutes if they did it right. The other ten percent are a mixture, of simply not doing it right, or having to leave it off overnight.

You may want to have her try again, or simply leave it off over night when she goes to bed.

If it keeps grabbing the same subnet, then either you're doing something that no one else is doing, or there's a very very very weird problem in her area, that no one else has seen before, and if that's the case, you'll want to contact Verizon direct.

I wouldn't recommend front line support for that, I'd say reach out to their twitter support or their direct support at the following link

Those are advanced technicians that would be able to help you better than going through the front line.

0 Likes