You’re a mean one, Mr. Grinch
You made our router fail!
No work can get done, and all the
employees are stuck and no emails can sent Mr. Grinch..
Oh What shall we do!

You’re a monster, Mr. Grinch,
You did so while I was Sick!
And now my backup person needs the password. Mr Grinch..
And I can hardly move!

You’re a vile one, Mr. Grinch
You made our website go poof on the Spot!
Critical systems are down! I’ve been called into town,
But luckily my backup Mr. Grinch, is smarter than you!

If you were wondering what happened to our website over this Christmas 2017 Weekend, on December 20th we had beginnings of a router failure with the complete failure on December 21st.  In moving over to backup router solution, we lost connectivity to our external website and a couple of other services, but the critical repository and internal network were back online.

Seeing it was the Christmas weekend, and a replacement router was scheduled to arrive on Tuesday December 26th I made the decision to just wait until the new router arrived on Tuesday instead of moving services to a different server then reverting the changes on Tuesday.

Being a small company, we have redundancy in several places in our system, but there are always single points of failures.  Understanding what those points are and how to work around them are Critical.  This was a piece of equipment that was in our control, but there are other pieces in the pipe that are property of the fiber supplier.  If one of these boxes fail, the response time can be significant.  

Today our business, like most businesses rely heavily on the Internet to function.  But as a small business we are a bit at the mercy of our ISP.  If they are down, we are down, and other than sitting on hold, with all their other customers, there is not much we can do about it.

We are lucky in that we have alternative ISP choices in our building.. (Driven primarily by us).  But the cost to have an “Always Available” backup line is significant.  

Although Mr. Grinch had his way this weekend with our website, our internal services were up and running fairly quickly this time..   And we’ve learned a couple of places where we ran into issues that could be improved.  

Until we do battle next time Mr. Grinch..

Happy New Years!