Downtime seems to have improved lately until we started having issues with one of our uplink providers.


3 out of 4 segments were down. We've opened a support ticket with them and working in conjunction to get this issue fixed.
Check back here for updates. This is an ongoing issue at this time, so if things don't seem to respond, packets don't seem to find their way to our network, this is why.
Servers affected: pepsi, moonshine, tequila, mail01 and mail02.
We'll keep you posted.

EN Support Team

Details on what happened to the core router. To be honest, I don't exactly know. It just won't initialize all network interfaces and that caused iptables to to be all screwed up.
Out of 5 currently used segments only 3 would come up in random order; very weird indeed.



After several power cycles, the problem persisted. We first thought it was the drive. After replacing it with a backup, the problem persisted.

Finally, after re-sitting the RAM it all went back to normal.

We'll keep a close eye on this bad boy and see what happens in the next few days.

Like always check back here for updates. :P

----

Update on this issue: 12/16/07
The problem was obvious yet frustrating. The router would not detect eth0 at boot time and shift all interfaces up one step. That caused all sorts of packet storms going down the other border routers and messing up the traffic flow.
We finally realized this after several hours and immediately fixed the issue.
Just FYI we've got a second core router setup and ready to take the load if any other issues come up along the way.
Things seem to work fine now.

Yesterday during the ice storm we had a power outage for about an hour.
All systems performed well on UPSes until we decided to go ahead and power down since it looked like this will take a while. We had to choose between waiting for the UPS power to run out and crash all the servers or properly shut down before the batteries drain out completely.
We went with option #2. Within the hour the power was restored and all systems are now operational.


Of course we had trouble with our core router which would not initialize properly after the power outage.
It ended up being a RAM issue which at least for the moment seems to have caused the problem.

10:00PM All systems are go.