Details on what happened to the core router. To be honest, I don't exactly know. It just won't initialize all network interfaces and that caused iptables to to be all screwed up.
Out of 5 currently used segments only 3 would come up in random order; very weird indeed.



After several power cycles, the problem persisted. We first thought it was the drive. After replacing it with a backup, the problem persisted.

Finally, after re-sitting the RAM it all went back to normal.

We'll keep a close eye on this bad boy and see what happens in the next few days.

Like always check back here for updates. :P

----

Update on this issue: 12/16/07
The problem was obvious yet frustrating. The router would not detect eth0 at boot time and shift all interfaces up one step. That caused all sorts of packet storms going down the other border routers and messing up the traffic flow.
We finally realized this after several hours and immediately fixed the issue.
Just FYI we've got a second core router setup and ready to take the load if any other issues come up along the way.
Things seem to work fine now.

[ view entry ] ( 1229 views )   |  permalink  |   ( 3 / 1764 )

---

Yesterday during the ice storm we had a power outage for about an hour.
All systems performed well on UPSes until we decided to go ahead and power down since it looked like this will take a while. We had to choose between waiting for the UPS power to run out and crash all the servers or properly shut down before the batteries drain out completely.
We went with option #2. Within the hour the power was restored and all systems are now operational.


Of course we had trouble with our core router which would not initialize properly after the power outage.
It ended up being a RAM issue which at least for the moment seems to have caused the problem.

10:00PM All systems are go.

[ view entry ] ( 6935 views )   |  permalink  |   ( 3 / 1535 )

---

<<First <Back | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | Next> Last>>