Hosting disasters

Every night is Halloween

Of course, there are many such memories from the early years of Level27, when things were not as professional as they are now. For example, an FTP account on a shared server was once hacked. But because of the permissions that were set on the servers at the time, this resulted in “dirty images” on all websites.

Another time, a server simply refused to restart. But when you only have two servers, that's obviously a major disaster. So every evening could be a bit like Halloween, with scary surprises. At that point, Maarten had had enough of Level27.

Data centre horror

Sometimes disasters are beyond our control, and data centres can play a major role in this. For example, Jeroen once received a call at 11 p.m. in Berlin informing him that an accident on the Brussels ring road had caused a power outage, preventing the generator in the data centre from starting. The result? All websites were offline. Unfortunately, that wasn't the end of it. Every few weeks, the power kept going out, and every day was spent anxiously waiting to see what would happen.

Thomas once witnessed 400 volts being applied to the circuits in a data centre in Vilvoorde. It is therefore no surprise that this did not end well for the data centre.

One of our competitors once had to drag a burning rack out of the data centre. A fire in the data centre is the worst thing that can happen to a hosting partner. We ourselves once experienced a fire extinguishing operation at Interxion. That all seems fine, but not everything can just keep running. Extinguishing with gas enters the room at enormous pressure and therefore has a huge impact on the environment. The irony? There wasn't even a fire in the data centre, just a technical malfunction caused by cables.

How can you prevent this? Of course, you can never be 100% in control (scary, right?). But it is very important to work with the right partners. Partners who work professionally and provide the right solutions should something happen.

The big storage crash

7 February 2013, a disaster that still gives Peter nightmares sometimes. A Thursday afternoon when suddenly nothing was online anymore. Everything was broken.

At the time, we were still running Solaris on our storage levels (ZFS in combination with iSCSI). At that point, we were reaching the limits of what these platforms could handle. And then one day, it finally happened. Due to a two-minute timeout, iSCSI concluded that the disk was gone and simply discarded it. As a result, everything became read-only. On top of that, there was a small bug that caused the data to become corrupted.

At that time, we were using Bacula as a backup solution. So we had backups of everything, but Bacula is a system that knows all the files on the server and copies them to another location. New virtual machines had to be set up and 70 VMs had to be restored from the backups. Peter and Thomas spent a whole night working on that.

Unfortunately, that wasn't the end of the story. The machines were restored on the surviving storage machines. These were newer and more powerful, so it seemed like everything would be fine. But no: on 14 February 2013, exactly the same thing happened again. This time, however, there were no survivors and all machines were impacted. That meant another night of working through the night.

This was a turning point for Level27. We couldn't continue like this; we either had to stop hosting or professionalise. To ensure that the disasters of the past were not in vain, we opted for the latter. We sought advice from other hosting providers in the Netherlands and ultimately chose FreeBSD with ZFS and NFS. At the time, it seemed unusual to choose NFS to set up virtual machines, but to this day, we have no regrets.