Many of our customers have asked the very reasonable questions of "Why was my website down for so long?" and "Why was it so hard to get a response from you?"
The short version is that all of our supporting web, database, email, and backoffice servers were impacted by an unannounced transfer of equipment and services with our former service provider.
All of our systems have now been re-provisioned with InMotion Hosting in California.
As you'll note from previous downtime notices and your own experience, this is not the first time that we had problems with our provider, and we had moved some of our systems to servers which we managed and controlled ourselves as a result. Unfortunately, this latest change impacted even the dedicated equipment which we were supporting.
Hostgator, despite the funny name, had been one of the most reliable providers since our original relationship with the company going back to 2006. Unfortunately, with the way of big business, good companies get bought by not so good companies, and ultimately their service has progressed up and down over the last few years. As part of their multi-year transfer process to consolidate their equipment with the new parent company, we were caught up once against in the latest incident which involved a change to the servers for all of our customers. The company actually denied making the change, and we had to hire an outside company to come in and do the troubleshooting. This process took almost 2 weeks, during which time the servers were up but running extremely slow, actually resulting in the appearance of a down server in some cases. We confronted the company, and they indicated it would be a month before they could reprovision the new hardware (apparently we weren't the only ones impacted). At this point we decided to move our webservers to a new company.
The move of the servers to a new company took another week, with our various customers having up/down time during the changeover, so that we could minimize data being lost on the old systems. Then we had the additional week to 10 days to reapply necessary security, spam protection, and other elements that are necessary to protect the operating environment. Finally we turned to getting internal support and business systems back up and running.
Unfortunately, our email servers, helpdesk, and even phone message routing systems were all hosted with the same provider. This left us basically completely shutoff from a communications standpoint. When we began the move we were even unable to move our domain information to the new servers because we couldn't access email, so this further complicated the process. Once the systems became unusable, the backlog on server emails, voicemails, and help desk tickets was in the tens of thousands...
We wanted to make you aware of this, and are providing this information not as an excuse, but only as an explanation. Dealing with websites, emails, and ongoing support means having to maintain the systems ranging from security, backups, upgrades and patches, malware, email spam, hackers and still be able to support customers. We rely on the companies that provide hosting of our systems to cover part of that work load, but in this case we became a victim of their success as they were acquired by another company without the same quality standards and commitment to their customers.
Friday, November 10, 2017