Tuesday, 30 November 2010

I miss snow days

One of the best things about working in a school was that when it snowed, quite often you got a day off work. Now, since I moved to HE, I have to come in and get soaked, cold and very tired from the even earlier morning commute.

I nearly could have done with a snow day yesterday, things nearly went tits up.

I was patching up some fibre in our main data centre when, all of a sudden, darkness. Okay, so the power has gone off. That is not a problem as the generator will kick-in... any-time... about now... no?.. bugger.

It became obvious from the lack of lights that the generator had not kicked in and we were running the whole building on a UPS which given the size of our building gave us about 20 minutes. Then phones, servers, the lot would go. Thankfully the doors lock open so we were not stuck in the basement comms room, but not sure how great that is for security (before you get any ideas we do have metal shutters out of hours).

We found out a contractor had been working in the plant room and for some reason had managed to knock everything off and stop the generator starting. So can we get into the plant room to look at fixing it? No as it is closed due to asbestos contamination and we need a mask, cover-alls, a risk-assessment and a key from estates.

We watched the UPS as it went down to 3 minutes left, 2 minutes left and so on. Grabbed some laptops to keep the switchboard up and running, forwarded all calls to our mobiles and grabbed some more laptops to take across the street and set up a helpdesk over there.

Then the UPS said went to 0% and everything was about to go off. Just as we got a message saying it was shutting down, LIGHTS, power, everything came back on. The contractor who was working on the electrics and had done 'something' to the cable had managed to get it back on.

Just in the knick of time, and people outside the building would not have noticed anything.

edit (from my boss): I saw two messages from the UPS, one saying 'I've had it', another saying 'I'm back', actually timed at exactly the same time (to the nearest second) - that's how close it seems to have been.

No comments: