When the Wheels Fall Off
Posted On: 2011-06-23
Ok so maybe this is the last time I ever write anything about hosting disasters. That's what I wrote about over on Porn Resource yesterday when I talked about the 4,800 Australian websites that have disappeared because a host got hacked and three servers were so badly damaged that nothing can be retrieved.
So what happens this morning? Our most important server ... and a whole bunch of other servers ... all died when a major power outage took out an entire data centre. I still don't have the full story but it seems that the power outage damaged some equipment and the backup generators didn't cut in.
When the sites did begin to come back online they were all very old versions ... but fortunately that didn't last too long ... just as we began to think that we would be spending the day doing nothing but uploading websites and databases and trying to work through a maze of new sites that weren't appearing at all ... everything came back to normal.
Fortunately for us it all happened at a time when there was very little traffic going to our clients websites and by the time our clients were beginning work for the day everything was back on line. Only two clients called about problems with their email and those issues were quickly sorted.
Of course that's all well and good for our clients but for us it was a very tense time. As I almost bragged in yesterday's Porn Resource column we have backups of everything ... but when old versions of sites began to appear and we couldn't FTP to the server to upload the newer versions ... well you begin to feel a bit sick in the stomach.
Not only do you begin to feel sick in the stomach but you tend to start to running around in ever decreasing circles ... ok so maybe you're cool, calm and collected under pressure but sometimes I do the running in circles thing. However my partner tends to get his problem solving hat on sooner than I do and he helped calm me down.
He suggested that we really couldn't do anything until we knew the extent of the damage and he reassured me that we do really have those backups that it looked as though we were going to need. He then suggested that we simply take a deep breath and wait till we knew what we were going to have to do and then spend a little time planning how we would attack the problem.
Fortunately everything came back online so we didn't have to do anything else but if nothing had been fixed we would have sat down and worked on a quick plan that would have involved identifying what we had to do and how to do it as well as prioritizing the work that had to be done.
Some sites needed to be up there quickly ... especially seeing that very old versions of the sites were showing. These were sites that were offering goods or services and prices that were well out of date or online shops where people were engaged in online advertising and couldn't stop their ads from appearing till their sites were back online.
Other sites on our server needed to be up quickly too because those sites are seasonal and we're moving into the time when lots of people will be looking at those sites for information. They were important but not quite as important as the first group.
And then there were sites that had a much lower priority. As long as those site owners could access their email the actual website itself could be left on the old version for a few hours.
That was how we would have planned to tackle the problem but we didn't have to. However it was a great reminder that we really do need to be prepared to handle the problems calmly and effectively any time the wheels do fall off ... as they did today.
So how are your planning skills? Do you know what needs to be done when problems with your hosting occur? Are you even able to remain calm when problems arise or do you waste time in panic and confusion?
If you are someone who struggles to remain calm when a crisis occurs perhaps you need to make your plans now and actually write them down. Keep a copy of all your access codes with the plan and don't forget to include a list of important phone numbers and the URLs of alternate service providers who can help you recover quickly.
For webmasters server failures are a certainty. No matter how good your host is it's not a matter of if a failure occurs but it is definitely a matter of when it occurs. So you need to be prepared ... you need to have a recovery plan and you need to keep that plan up to date.
When things are running smoothly a plan like that seems unnecessary but when disaster strikes and the wheels fall of you will be glad you have it.