Maxim_Kazmin - Fotolia
Registrar and web hosting firm 123-reg has admitted that it has ‘effectively deleted’ an unspecified number of websites.
The ‘error’ occurred on Saturday morning. The company, which hosts 1.7m sites in the UK, said that a clean up script was run to show the number of machines active against the firm’s master database. The faulty script showed 'zero-records' for a number virtual private servers that were in fact live, deleting what was on the host.
The UK’s largest registrar said that it is now using data recovery tools in an attempt to recover packages ‘bit by bit’.
“Due to the nature of this process, we are unable to prioritise the recovery of specific hosts, but we are focused on restoring bulks of data at the moment,” the firm said in its most recent update. “Progress is visible, and we have already managed restoring a certain number of hosts. However, we are unable to estimate when will the recovery be finished on all of the affected packages.”
While 123-reg has not specified exactly how many sites have been affected, a torrent of angry customers have taken to social media to express their outrage.
Luke Spencer is managing director of Digital Save, a web-based mobile phone and audio accessories business that was affected by the crisis.
“I noticed the issues on Saturday morning when I logged into our website and the it didn't load,” Mr. Spencer told MicroScope. “My heart sank as we had a similar issue in February when 123-reg had a sudden power outage and our site was offline for five days.”
“123-reg's handling of this issue has been diabolical. Their support page has been updated infrequently and the information provided has never given the full facts.”
Initial updates on the company’s support page said that there were “connectivity issues” and it was not until Sunday evening that 123-reg issued an official response explaining what had happened (full letter below).
Mr. Spencer hasn’t wasted any time in signing up for a new VPS with a different provider.
“We are in the process of setting this up and we should hopefully be back and selling again by Tuesday. We will never return to 123-reg and we would not recommend them to anyone due to the multiple issues we have had with them.”
The MD said that while his company had offsite backups, he was aware of other businesses that used 123-reg's inbuilt backup system, which was also affected by the rogue script.
“This whole situation has been an awful experience for us. We are a small business and we are finding it tough trading at the moment," the frustrated business owner told us. “We estimate we have lost around £7,000 of sales over the weekend and around £2,00 today. Overall, this will have easily cost us £10,000 in lost sales. As a small business, we will find hard to cope with this loss of revenue.”
123-reg has advised customers to ‘self-rebuild’ using local backups where available.
Kaya Ismail, editor of CMS Critic, published a article days before the wipe-out occurred, warning his readers to stay away from 123-reg.
"I have had my reservations about 123-Reg for a while, which prompted me to write about why people should avoid them, just days before this disaster struck,” Ismail said.
“However, I never imagined that 123-Reg would make such a large-scale mistake. I don't expect that many people will be trusting them to handle their websites in the future, and rightly so.”
“My sympathies go out to the customers who have been affected, and I advise them to migrate to more capable web hosting companies."
123-reg was not available for comment at the time of publication.
Since this article was published, 123-reg has provided MicroScope the following statement:
Letter to customers sent on Sunday
I am writing to you to explain what happened to some VPS services on 16.04.16. This email is to detail what our steps have been. I am committed to open communication with all customers and would like to take this opportunity to explain in detail.
So what happened to some services? As part of a clean-up process on the 123-reg VPS platform, a script was run at 7am on 16.04.16. This script is run to show us the number of machines active against the master database.
An error on the script showed 'zero-records' response from the database for some live VPS. For those customers, this created a 'failure' scenario - showing no VM's and effectively deleting what was on the host. As a result of our team's investigations, we can conclude that the issues faced having resulted in some data loss for some customers. Our teams have been and continue to work to restore. What have we done? We have been working with an extended team of experts and have left no stone unturned. Our teams have been working long into the night to restore as much as we possibly can. We have also invested in external consultants to recover, in the best way possible.
We have recovery running on the VPS servers and some are restoring to new disks. We have also begun copying recovered VPS images to new hosts and we expect some VPS to be back up and running throughout the night and in to tomorrow.
Our teams have worked for more than 24 hours and will continue to do so. No stone is being left unturned.
As the technical teams come back with updates for individual VPS we will communicate updates to customers.
For those customers with their own backup of their settings and data, if you wish to restore services yourself you can do this by issuing a relmage command through your 123 Reg control panel, this will give you a freshly installed VPS on a new cluster, where you can restore your service.
I understand that some customers may have lost some confidence in the service that we offer. So, I want to explain what we have done to prevent this happening again. We have started an audit on all cron-jobs and scripts controlling the platform, and associated architecture, so that no script will have ability to delete images, only suspend. For image deletion for those suspended over 28 days we will have a human eye to double check. A new platform will be available by the end of the year for customers which we will provide self-managed and automated snapshot backups, in addition to architecture technology to backup the whole platform, something that is not available on the current platform. I hope this goes some way to win back your confidence.
123 Reg Brand Director