Servers went down. We think it had something to do with the network hardware and / or the drivers ( Atheros AR81Family ) and their compatibility ( or lack thereof ) with our Centos 5.5 boxes.
As our servers are in the PRC, I can’t physically touch them – luckily we have an excellent team of capable admins who can. After being alerted that the servers were down, they went to the datacenter and managed to convince the machines to work.
I’d noticed some similar issues with our development server ( same hardware ). The server crashed quite often when using rsync to backup directories with a large amount of small files, and would need to be hard-reset. Not cool.
So I spent some time tracking down the driver download page – it’s hidden :( and updated our network drivers. We were 5 versions behind! Here it is:
I’ve spent the day trying to put the development server under as much stress as possible, using ab, flood and rsync. So far it hasn’t crashed, which is a good sign.
I had a little trouble installing flood on my work machine – ran into errors with Macports installing db46, required for flood.