Slows Speeds & Packet Loss on Sprint Network

BurstCEO · #1 03-27-2003, 13:11

Currently we are seeing slow speeds and packet loss on the Sprint OC3 line. This is starting several hops before our network, and speeds up once arriving to our network, though no where near as fast as it should be...

Traces thru Levels3 & Verio seem normal.

We are contacting Sprint, and hope to have resolution shortly.

Example 1:

6 sl-gw4-roa-0-3.sprintlink.net (160.81.15.21) 16.592 ms 16.665 ms 16.697 ms
7 sl-bb20-roa-2-3.sprintlink.net (144.232.17.201) 20.605 ms 16.996 ms 16.687 ms
8 sl-bb24-pen-10-1.sprintlink.net (144.232.8.141) 104.378 ms 148.371 ms 157.617 ms
9 sl-bb20-pen-8-0.sprintlink.net (144.232.5.165) 154.386 ms 167.764 ms 163.508 ms
10 sl-gw1-pen-9-0.sprintlink.net (144.232.5.2) 177.287 ms 169.418 ms 212.677 ms
11 sl-hostnoc-1-0.sprintlink.net (144.228.171.178) 50.703 ms 50.281 ms 50.610 ms
12 66961928.hostnoc.net (66.96.192.8) 51.090 ms 50.937 ms 50.593 ms

Example 2:

7 204.255.174.182 (204.255.174.182) 21.852 ms 19.308 ms 20.666 ms
8 sl-bb26-rly-15-3.sprintlink.net (144.232.19.178) 30.804 ms 40.868 ms 30.881 ms
9 sl-bb24-pen-12-0.sprintlink.net (144.232.20.110) 128.675 ms 218.450 ms 220.544 ms
10 sl-bb20-pen-8-0.sprintlink.net (144.232.5.165) 43.309 ms 44.730 ms 47.559 ms
11 sl-gw1-pen-9-0.sprintlink.net (144.232.5.2) 44.612 ms 45.040 ms 48.535 ms
12 sl-hostnoc-1-0.sprintlink.net (144.228.171.178) 67.151 ms 66.946 ms 63.574 ms
13 66961928.hostnoc.net (66.96.192.8) 63.145 ms 61.266 ms 63.945 ms

SMA
PRES/CEO
BURSTNET

BurstCEO · #2 03-27-2003, 18:41

Problem has been determined to not be related to Sprint specifically.
We have turned all lines back on for now, AT&T, Level3, Sprint, & Verio.
The problem is NOT due to a lack of bandwidth. We had these down one at a time today to see if perhaps one line or router card was the culprit.

At this point we cannot find the problem. It is not consistant, and is completely intermittant. 90% of traceroutes are fine, then boom a bad one hits. We cannot find flooding of any kind either internal or external.
The router is reporting no errors, and is not overloaded. We have completely written a new router configuration file, and that has not changed results at all either.

We have brought in outside experts to assist with the issue, and they cannot find the problem either.

We have no further information available at this time. We are continuing to troubleshoot the issue, and will post information here as soon as more is available.

SMA
PRES/CEO
BURSTNET

BurstCEO · #3 03-27-2003, 19:08

Router reboot at 6:00 PM EST

SMA
PRES/CEO
BURSTNET

BurstCEO · #4 03-28-2003, 00:35

Backbone speeds are currently fine, and have been for some time. There are problems with packet loss at the gateways, and this is what we have been unable to correct up to this point. Still working on the situation, with no ETA for correction. Will post update here as soon as available.

SMA
PRES/CEO
BURSTNET

BurstCEO · #5 03-28-2003, 01:31

Updating to newest IOS on router.
Further updates shortly.

SMA
PRES/CEO
BURSTNET

BurstCEO · #6 03-28-2003, 11:15

IOS upgrade has not corrected the problem.
We are adding in additional routing equipment between the main router and the core swicthes to reduce load on main router to see if that corrects the problem. This wil takes 2-3 hours to get everything configured and switched over.

Outside consultants have stated the possibility exists that even though the system reports CPU load as perfectly fine, it is possibly spiking every few seconds when BGP/ARP refreshes since the network is so large now.

If this does not work, we will be going out an purchasing completely new equipment, te replace existing routing equipment, as that is really the only other vaible solution now, as we have troubleshooted every possible cause.

SMA
PRES/CEO
BURSTNET

BurstCEO · #7 03-28-2003, 15:04

We believe we have found the problem, and are rectifying such now.
Looks to be at the switching level.
Details to be posted here as soon as issue is corrected.

SMA
PRES/CEO
BURSTNET

BurstEmployee · #8 03-29-2003, 00:50

This has been corrected. We had several switches die simultaneously. We replaced these and at this time all problems appear to be corrected. We thank you for your patience during this trying period.

BurstCEO · #9 03-29-2003, 01:57

<< We had several switches die simultaneously >>

And in a manner NOT recognizable...as they were still functioning, and showed no signs of being the cause. When we pulled one switch at a time, to fnid the cause, we could not, as it turned out to be more than one. Also, had to replace cross-over cables on said switches, and remove 1-2 servers involved in cause at the same time, that were on those switches.

We also upgraded the IOS on the core router, and wrote a completely new configuration file.

Service seems back to normal since this was resolved earlier afternoon Friday, however, we did see some minor packet loss and slowdowns during 5PM/6PM EST peak traffic periods. We believe this to be a separate issue from the switches malfunctioning, but a contributor to the overall recent service drop. This has been diagnosed as our core router becoming overloaded as ARP and BGP refresh every so often, and inconsistantly causing a CPU spike every so often. Not enough to hit the MRTG monitoring graphs, but enough to cause these issues. The reason this would start occuring now, is due to the size of the network, with so many MAC addresses and IP addresses now on the network.

Resolution: We will be installing additional routing equipment to handle some of the processing, mainly ARP, and relieve the core router(s) of almost everything but BGP.

Also, the overall network will be getting re-segmented over the next 4-6 weeks. This will confine internal flooding to specific switches or possibly even a single offending machine, and not allow such to affect the entire network. More details on this to follow shortly...

SMA
PRES/CEO
BURSTNET