Incapable residential DSL support? Just lost Verizon a potential 250k/yr in business accounts
jp78945
Enthusiast - Level 2

Has anyone found a way to work through the incapable layers of support to find the responsible, talented workers I know exist at Verizon (someone with at least an entry Cisco/networking cert)?  I've opened at least five tickets with Verizon this year alone on frequent disconnects, poor throughput (1 - 3 Mbps, when 12 - 15 was achievable prior to January) and can't seem to reach anyone willing to "own" my problem long enough to fix it.  In fact, every supervisor or technician has closed the ticket despite my telling them my problem is not resolved.  <end rant, sorry!>

Speed tests still (on occasion) look ok, but sustained downloads hover at 3 Mbps at best.  According to the a supervisor in DSL support, this is ok and "the best we can do."  The third technician who visited my house didn't understand that 300 KB/sec download is very poor, considering months ago it was at 1.5MBs (12+Mbps), but was also on the phone with his instructor for much of the visit.  I live 2 blocks from the central office and have had no issues until this year (heavy rain after a dry year), and modem syncs are very high.  Another recently used modem loses sync up to 50 times a day; I believe the latest I'm using just masks my problems a bit better.  I do see 20 - 30 HEC errors on occasion.  However, TCP retransmits are EXTREMELY heavy on the last leg of my connection (I'd assume the DSLAM over ATM to our CO and then my residence, I think).  Enterprise level testing software sees the same (as does the free test at visualware.com).  Jitter is often very high.  Verizon was "kind" enough to disconnect the rest of my home wiring to help isolate the problem, but it persists even with all wiring disconnected from the NID and my modem plugged directly to the NID test port.  I originally thought it was congestion, but starting monitoring via professional tools and noticed it happens all hours of every day.  Anyone with ideas... let me know!  I am even willing to send modem logs and test data to those with thoughts.  I hold multiple Cisco certs so am happy to talk nitty-gritty with any community members with ideas on how to further isolate the problem.

Unfortunately, Verizon is going to have to learn the hard way that dumping their support to foreign call centers is not a sustainable business plan.  As head of IT infrastructure for a billion dollar business, and consultant for many more, I can say that Verizon just got dropped from a (quite lucrative) table.

Re: Incapable residential DSL support? Just lost Verizon a potential 250k/yr in business accounts
smith6612
Community Leader
Community Leader

To be honest, the Business DSL connections are backed by some better support. Two different queues. The same with Telephone. I've dealt with Business support for other businesses in the area, mainly on the Telephone side, and they have been helpful, besides one or two no tech shows which happens for any company.

I myself, also work for a very large Internet-based corporation. It's a company with a name you're probably familiar with, and I, too am in the IT Department. I'm not a Head IT individual, but I am in a datacenter dealing with requests and issues on a day to day basis, and dealing with Juniper, Cisco, Unix, Windows, you name it on a routine cycle. If a connection in the datacenter starts to even develop the slightest problem, every alarm starts to go off and it's essentially an all hands on desk to get the problem resolved. After all, if stuff doesn't work as it should, the Internet will certainly notice, and then we'll have everyone breathing down our necks wondering why stuff is broken.

Anyways, to troubleshoot this issue, we will need a Reverse Traceroute from another computer to your connection, and we will also need your DSL Line Stats/Transceiver Statistics. I'm looking for a few things here to see why your speed is lousy, and to also see if you've been placed on gear you should not be on. Once we do this and get the info we're looking for, I'll get you to someone who will take care of this, right past the Level 1 tech support nonsense. If you want to post up the rest of the information from the software you used to analyze the problem, I'd be glad to take a look at that too.

If you're wondering why I stated we need to check the equipment, and why I will send you off to someone, I can tell you that a few weeks ago, Verizon had moved me onto equipment which I should never be placed on, due to the equipment out in the field. Right now, I'm on an Alcatel Litespan 2000 Remote, which, if you're not aware of them, are pretty much basic ADSL, compact DSLAM units which are driven by OC3 lines, perhaps muxed down to T3s per DSLAM Bank and use Alcatel ATM Switches. Verizon does not provision speeds higher than 3Mbps out of them, and even if a line does slip higher, the PVC does not permit the speed to go past 3Mbps regardless. For the sake of things, other remotes that Verizon uses are Catena remotes, which are about the same age as the Litespans, but have OC-12s running out to them, and banks of cards are muxed down to T3 lines per remote. Once again, remote is limited to 3Mbps max. Then there are newer Adtran remotes, which I'm not super familiar with but can be driven by anything from Several T1s to Gigabit Ethernet over Fiber, and these are the ones that you'll most likely find running ADSL2+ and maybe pushing higher speeds.

So anyways, away from the field end of things, back at the Central Office, Verizon has banks of BRAS/Edge routers which serve as the gateway between the Backbone and the Internal ATM network within the CO. You'll find that the Verizon COs for the DSL network have one or even both of the following router variety available: Redback Networks/Erricson edge router (do not know series), or Juniper ERX series. The Junipers are the faster, meaner routers. Redbacks are the legacy units that have been in service for ages. One key thing to keep in mind, going back to the whole remote type thing, is that the Redbacks are the only edge routers that should ever be used on the Litespan remotes. The Juniper ERX Routers are incompatible with the Litespan ATM switches, and this as a result, causes all sorts of issues with lines. Most notably, you'll find packet loss that really isn't there, download speeds are in the toilet, and upload speeds are right where they should be. You'll also find issues on self-tuning Operating Systems where Wireshark or other packet sniffing software or even hardware can see a machine's Receive Window growing and shrinking a rediculous amount.

Well, during the middle of the night, around 2PM when maintenance is normally performed, my line had lost it's logical link to the Verizon network. I was sleeping at the time. Upon waking up, my PPPoE connection was down which I forced to re-connect. It had to have been down and not automatically reconnected for some reason, so one of the first things I did was check the subnet and default gateway I was on. First thing I notice was my Default Gateway to Verizon was entirely different. 10.41.21.1 . I've seen that IP before, and it smelled of Juniper. I then went to perform a speed test. Page load speed was rediculously slow. So I fired up download and upload to see how the line was doing. The download? It started at 128KB/s, and then immediately became inconsistent, causing my overall throughput to bomb below 500Kbps. My upload was perfectly fine. With that being said, the connection was barely usable. I couldn't play any online games to my own dedicated server as it would choke out due to the "packet loss" that was being caused. Downloads often were very slow if they didn't stall out, and web browsing was iffy.

I got this problem resolved by using the card I always pick when network issues arise. Go to my Internal contact deep within Verizon who can dig deep into my line, find the problem, and fix it, no questions asked. Gave him the only piece of info he needed, and before I knew it, he had placed me right back on the old Redback I was on before, and my connection was only down for a good 5-6 seconds. No one noticed the PPPoE session dropping and reconnecting. Now here I am with a line that runs as it should.

So with that being stated, that's one of the things I'm looking for. If you actually had 10-15Mbps in the past, you for sure should be on some nice DSLAM gear, perhaps an Adtran or an Alcatel Ultra Density DSLAM. 10-15Mbps lines are required to be on Junipers due to the Redbacks lacking the capacity, so this is another thing I'm keeping in mind.

Re: Incapable residential DSL support? Just lost Verizon a potential 250k/yr in business accounts
Hubrisnxs
Legend

Try

Verizon Direct

These are high level technicians that work on customer reports through a secure forum where only you and the verizon rep can see your post.

0 Likes
Re: Incapable residential DSL support? Just lost Verizon a potential 250k/yr in business accounts
jp78945
Enthusiast - Level 2

Good info, and especially nice to learn a little about the DSLAM tech! We "enterprise" guys see very little of that, so for fun, I may have to start spending some more time with one of the independent (small) telephone companies I sometimes do work with.

Here's what that DSL modem stats look like:
DSL Mode: Auto
DSL Negotiated Mode: ADSL2+
Connection Status: ShowTime
Speed: 17647 / 1183 Kbps
ATM QoS class: UBR
Output power (down/up): 12.2 / 20.6 dBm
Attainable Rate: 22536 / 1271 Kbps
HEC errors (down/up): 0/19 (never more than 200 in 24 hours)
OCD errors (down/up): 0/0 (sometimes I do see a few of these guys, but nothing crazy)
LCD errors (down/up): 0/0
SNR Margin (down/up): 9.1 / 6.4 dB
Attentuation (down/up): 13.5 / 4.3 dB
However, things appear to be working slightly better this morning. This is what I typically see in my tests:
Download speed: 4827 kbps <------
Download consistency of service: 8% <------
Maximum TCP delay: 991 ms
Minimum round trip time to server: 23 ms
Estimated download bandwidth: 11020 kbps
Download TCP forced idle: 0%
TCP MTU: 1500
Bytes dup: 0
Fast Retransmits: 3
Bytes retransmitted: 70080 <-----
Dup acks: 74
Persist timeouts: 0
Upload speed: 1018 kbps
Consistency of service: 99%
Packets retransmitted: 48

The last time I disconnected all of the house wiring at the NID and connected a test set to the test port, I swear I heard a slight, continuous, wah-wah sound (the technical description eludes me, so forgive the musical jargon!!).

I'll try to perform another traceroute this evening, but in any previous test, all looked fine except for the last hop (my default gateway to my modem). There is a request in to change the copper pair to my residence, so if anything, that'll rule out one possibility (if it actually happens). They should visit within 48 hours or so; I'll send an update to interested parties after that is complete.

PS. Thanks for all who have provided additional paths for assistance, including those who sent the PMs. It sounds like Verizon is more than willing to pick the problem up again and give it another go.

Re: Incapable residential DSL support? Just lost Verizon a potential 250k/yr in business accounts
weedeater1
Contributor - Level 3

Personally, I would think a SNR that low would be a real problem.

Re: Incapable residential DSL support? Just lost Verizon a potential 250k/yr in business accounts
smith6612
Community Leader
Community Leader

I don't see this as an SNR problem immediately. His copper pair actually looks pretty good. He's got the full 15/1 sync rate, he's got very little errors, and his line is reporting if it it's really pushed it will do very close to the max of ADSL2+ of 24Mbps down. I'd say it's a very healthy copper loop, so do not get the copper loop changed out yet! An SNR below 10, while it isn't ideal, doesn't hurt much if the line has been proven to remain error free or give the rare error and also retain sync and Quality of Service. My Frontier line is a connection that is running a little over 2 miles of copper. I'm on the oldest possible gear I could ever be on, and the upload speed for the card in the DSLAM I'm in maxes out at 896kbps. Since right now me and a tech are troubleshooting a problem with the line, I've actually got my upload uncapped from 448kbps all the way up to that full sync rate. I've been holding there is 6dB margin and no errors over two miles of copper at the max speed of ADSL for upload. No problems what so ever with that line holding an upload 24/7, or download speed when it isn't suffering from congestion at night. The line's as tough as a rock, and yes it has been hit by lightning a few times 😄

I really vest the problem is closert to the network side. If speeds are better in the morning it means gear or a link is faulty or is overloaded. I'm still waiting on the reverse traceroute, if you could please provide that since I need to see if you're on Juniper or Redback, or some other gear. Also, I do want to confirm you are not on Windows XP or some other OS that does not tune the TCP/IP Stack, right? Basically, anything Windows Vista, Windows 7, or Unix based is what I'm not super concerned about.

The most solid way to determine if the problem is a loop/noise issue would be to simply have someone with deep enough access to the DSLAM and PVC for the circuit to specify a custom rate for the line. Setting sync to 14Mbps down, 900kbps up will bring the line to a level where the SNR should be 10dB or higher in both directions to satisfy any technician. If things improve at that speed or at 10Mbps/1Mbps or even 10Mbps/768kbps, then we can proceed with troubleshooting the copper loop. But otherwise, it looks solid to me at it's current 17Mbps/1Mbps sync rate.

Re: Incapable residential DSL support? Just lost Verizon a potential 250k/yr in business accounts
dslr595148
Community Leader
Community Leader

@Smith6612 wrote:
I'm still waiting on the reverse traceroute, if you could please provide that since I need to see if you're on Juniper or Redback, or some other gear.

You mean this..


Visit http://www.giganews.com/line_info.html and post up the Traceroute the page shows, if you wish. Be aware that the final hop (bottom-most line of the trace) will contain a hop with your IP address in it. Remove that line. What I'm looking for is a line that mentions "ERX" in it's name towards the end. If for some reason the trace does not complete (two lines full of Stars), keep the trace route intact.


??

0 Likes
Re: Incapable residential DSL support? Just lost Verizon a potential 250k/yr in business accounts
smith6612
Community Leader
Community Leader

Pretty much 🙂

Re: Incapable residential DSL support? Just lost Verizon a potential 250k/yr in business accounts
jp78945
Enthusiast - Level 2

Hey guys,

Sorry for the long hiatus; following a pretty bad auto accident and several weeks of troubleshooting by Verizon executive support and maintenance teams, the determination was "network congestion."  I did a bit of surveying in the community, and unfortunately found only one other individual with the 15 Mbps provisioning.  He also experienced some of the trouble that I did through the past few months.  Interestingly enough, I have access to several other DSL connections in the area provisioned for 3.0 Mbps that have been absolutely fine through these peak hours when I experience trouble.  For example, the 3.0 Mbps provisioning will knock a consistent 2700 Kbps during a period when I will see 1.0 - 4.5 Mbps inconsistently.  That makes me think that the true problem still lingers :).

In what was definitely a thorough attempt by Vz guys, at least the following was done:

- New NIU

- New drop from pole to residence

- Different interface at CO

- Different DSLAM

- (supposedly) different copper pair

- Over a week of periodic troubleshooting by one person at a Verizon maintenance facility

As requested, here is the reverse traceroute...

1 gw1-g-vlan201.dca.giganews.com (216.196.98.4) 0 ms 0 ms 0 ms
2 g3-0.bb1.dca.giganews.com (216.196.96.62) 0 ms 0 ms 0 ms
3 xe-8-0-7.ar2.iad1.us.nlayer.net (69.31.10.97) 3 ms 1 ms 3 ms
4 ae4-40g.cr2.iad1.us.nlayer.net (69.31.31.185) 2 ms 0 ms 0 ms
5 xe-3-3-0.cr1.atl1.us.nlayer.net (69.22.142.105) 13 ms 23 ms 13 ms
MPLS Label=411395 CoS=1 TTL=0 S=1
6 xe-2-2-1.cr1.dfw1.us.nlayer.net (69.22.142.25) 33 ms 33 ms 37 ms
7 ae1-50g.ar1.dfw1.us.nlayer.net (69.31.63.126) 43 ms 35 ms 35 ms
8 TenGigE0-0-2-0.GW5.DFW13.ALTER.NET (152.179.51.41) 48 ms 45 ms 46 ms
9 0.xe-5-1-2.XL4.DFW7.ALTER.NET (152.63.99.105) 46 ms 47 ms 52 ms
10 0.ge-3-0-0.DFW9-BB-RTR2.verizon-gni.NET (152.63.3.234) 50 ms 0.ge-6-3-0.DFW01-BB-RTR2.verizon-gni.net (152.63.96.50) 56 ms 53 ms
11 as0-0.DFW01-CORE-RTR2.verizon-gni.net (130.81.20.63) 54 ms 53 ms 51 ms
12 A9-0-1711.HERNTX-DSL-01.verizon-gni.net (130.81.135.61) 57 ms 57 ms 57 ms
13 pool-xxx-xxx-xxx-xxx.herntx.dsl-w.verizon.net (xxx.xxx.xxx.xxx) 70 ms 67 ms 66 ms

An interesting find through working with Vz: when performing an extended bandwidth test, I see a TCP pause about ever 5 - 10 seconds.  During that period, all throughput plummets, and then starts again.  The pause is up to a second long.  My guess it is this pause that causes TCP windowing to work its magic, which affects sustained throughput and absolutely destroys the chance of streaming any media.  No idea if this is related to the frequent sync failures I saw with my previous modem.

As perverse as this sounds, figuring out the root cause of these issues is turning out to be quite entertaining.  I've considering contacting the exec support folks again, but before doing so, thought I might get your opinion once more (as it helped while working with Vz) and maybe even some other avenues suggested above.

0 Likes
Re: Incapable residential DSL support? Just lost Verizon a potential 250k/yr in business accounts
smith6612
Community Leader
Community Leader

Ouch on the accident. Hope everything's been alright since that happened and everyone is feeling better!

Verizon's network is ultimately pretty expansive. You'll find times where some lines will experience congestion whereas others will not. All depends on what gear they are going through at the Central Office and any possible remote, hence why some lines can push 3Mbps all day, all night and others absolutely die at night.

Are the TCP Pauses you're seeing consistent with one another, as in, do they happen at a set time? Also, do these pauses occur regardless of how much utilization the line is seeing? Are you encountering any cases of packet loss when this happens or is it a simple stop and go transfer? It might be something related to MTU if you're seeing excessive fragmentation. It might also be a line issue if for some reason your line gets hit with a spike of noise, making uncorrectable errors occur which can cause these.  Of course, then there's the network end of things that might cause some issues.

Try running an NPAD test and see if it is able to detect anything abnormal, such as packet loss in transfers or an excessive amount of pausing.

Do a test to these two servers:

http://npad.iupui.mlab2.lga02.measurement-lab.org:8000/ (New York CIty).

http://npad.iupui.mlab2.nuq01.measurement-lab.org:8000/ (Mountain View, California)

Ping the above servers to determine the correct RTT to fill in. The value you get for latency should be increased by about 5ms and then filled into the NPAD Application. On a 15Mbps DSL Connection with Verizon, please set the Target Rate to 14Mbps. 10Mbps lines should be set to 10Mbps. When the results page appears, please copy/paste the edited results here as the results will contain your Hostname and IP address if linked directly