RESOLVED. The source of the problem (a Denial of Service attack) was traced and has been resolved. ISRS, eServices and D2L systems were not affected, however, users at Metro schools experienced problems connecting to any website.
Details:
(1) A Denial of Service (DOS) attack was launched against an IP address at one of our campuses. The attack was about 100,000 packets/second, minimum packet size, UDP port 80, directed at a single host on a campus. This is not an unusual event, and there is no indication that the campus did anything to cause the DOS attack.
(2) Routing the packets from the DOS attack caused a router CPU's to overload.
(3) The router with high CPU stopped talking to its neighbors.
(4) The neighboring routers stopped sending packets to the overloaded router. (This is by design).
(5) The routing protocol re-routed the traffic around the overloaded router to redundant routers.
(6) The redundant routers that took over the load from the overloaded router became overloaded from attempting to route the DOS attack to the campus. Go to (1), repeat (1) -> (5) at 3 minute intervals.
(7) It took us well over an hour to figure out that this was a DOS attack.
(8) Once we figured out it was DOS related, it took some time to get mitigation in place.
The DOS attack is normal. We deflect DOS attacks pretty routinely, and we have deflected attacks larger than this. The attack should have taken the affected campus off line. Both campus router and firewall should have been overloaded, but the rest of the metro area should not have been affected.
We saw the 100k packets per second, and OET suggested that this was a DOS attack. I dismissed the suggestion, assuming that 100K packets per second was not sufficient to cause an overload of the particular routers. We instead focused on other routing and network issues. That cost us over one hour.
Once we determined that this was a DOS attack, it took some time to work our way upstream to a router that could block the attack without causing more routing issues. Blocking an attack can overload the router that attempts the block, and the upstream routers are used by all state agencies, so we were pretty cautious when implementing the block.
We are following two threads to reduce the impact of future events like this.
(1) OET is looking at the affected routers and their ability to route at rates similar to the DOS attack. They are also looking OSPF configuration to see if they can make the config more resistant to CPU overload.
(2) We will look at the tools that we already have an see if we can come up with a simple way of detecting DOS attacks sooner. Early detection gives us a better chance of minimizing the effect.
(3) We'll make the decision on the best method to deflect or block the DOS earlier in the incident. We have several places to block the attack, and we need to decide which ones will be most effective for a particular type of attack without causing larger problems.
--Mike
___________________________________
Michael Janke
Director, Network Services
Minnesota State Colleges and Universities
Wells Fargo Place
30 7th St. E., Suite 350
St. Paul, MN 55101-7804
Voice: 651-556-0583
Fax: 651-649-5770
Cell: 612-964-3340
