One year ago, about 150 of our servers were located in a data center near Boston. Thousands of our customers are based in the USA, which is why we have servers there (our company is actually based in Bulgaria, Europe). New hardware developments require regular upgrades – we do not wait for our servers to “burn out” before we change them. I often compare our business to what airline operators do – we never make compromises with security and reliability, which is why our servers do not “crash.” Yes, you can fly from A to B for $290, but you could also do the same for $29 – especially if you are not concerned with troubling safety statistics.
Since the very start, which was back in 2001, we keep one spare server for every five active ones. These machines are always available in the event of an incident. And even though we rarely resort to using them, they age and are upgraded with newer ones, too. Through the years, we have spent hundreds of thousands of dollars for spare servers, which end up scrapped without ever being used, but this is a justifiable expense.
I still have vivid memories of a time in 2003 when we saw a sudden “wave” of over one thousand new customers who told us how their provider’s server burned down, and they had already purchased a replacement, but it was not going to come for another two days.
Several months ago, we had to make another massive hardware upgrade. Not only do such changes to our infrastructure cost hundreds of thousands of dollars, but they also involve business trips to the USA or Hong Kong, during which some of our colleagues are away from home for several weeks, and the upgrade procedures follow a strict procedure and plan.
This is the exact moment when we received a letter from the company that operates the data center near Boston. The letter said that this data center will be decommissioned in a year, but the company has built a new one in the immediate vicinity, and they were hoping to keep us as their customer. Who would choose not to, when our monthly bill (rent for physical space, electricity, security, air conditioning, Internet access) comes to $38,000, and we have been paying our fees on time for 15 years?
This would have been very bad news at any other time, but not then! Shutting off all of these servers even for a minute would be unthinkable, and moving them physically – even to the building next door – would take at least 30 minutes, with additional risks involved.
I do not like compromises, and I have some experience with physically moving servers: back in 2004, another service provider in the USA did not comply with our standards for quality, which is why 24 servers had to move between states. To achieve this, we bought 24 new servers, deployed them to our new location, and we migrated the data online. The thousands of our customers hosted there did not see any downtime during the process, after which we went to collect our now unusable 24 machines from the old provider.
However, the same news in the middle of a current upgrade planning process meant that we wouldn’t have to replace anything down the road. We just had to make a copy of our current setup at the new data center building, and then transfer the data “on the go”, without any discomfort for our customers and their sites’ visitors.
Our provider was not aware of our upgrade plans, so we duly informed them of the great inconvenience that we were being put in, including the huge expenses that we had to make because of their decision. We also had to let them know that since the move was inevitable, we also had to consider alternative providers – and they are not one of the cheaper ones. They became worried, so they offered us a better deal on traffic and collocation – saving us $14,000 every month for a number of years. In the long term, this will repay our current investment in new hardware. We have been in business for 20 years, so numbers like this one accumulate. We seized the moment, and we signed a new long-term contract with the new, better parameters. For example, the new price for Internet traffic is 4 times lower for us, which will be reflected in the plans that we offer to our end clients and resellers.
We came up with a new action plan, which started with purchasing latest-gen hardware for almost a million dollars: servers, based on Dual Intel Xeon Gold CPUs, Intel data center-grade SSD drives, 256 GB of DDR4 memory, 10 Gbit network switches, smart PDUs, and everything else, with the best possible features. Due to the improvements in hardware and virtualization technology, our new servers in the Boston facility will go down from 150 to only 30 – this did not save any money for the hardware, but it will save on data center space, as well as on the huge amounts of electricity required to power such hardware, which is not only a saving for our budget, but also for the environment.
Ultimately, we made a full copy of our current infrastructure – only much better – in the new building, and after every piece of equipment had been fully tested to our standards, we began the long process of migrating tens of thousands of client sites – millions of files, thousands of terabytes of information. This was not felt by our customers in the slightest. Once the operation had finished successfully, we realized that there is hardly another hosting company out there with a single data center filled only with all-new, top-of-the-line servers.
In addition to this, and thanks to choosing the best possible components, we ended up using only about 1/4th of the new servers’ capacity. In turn, this suggests that we should discontinue our 18 years of marketing silence, and invite thousands of new customers to utilize the well-oiled machine that we have built in this time. Which will be beneficial to everyone – to our clients, to us, to our country. Well, to almost everyone…
This morning, I had to decide what to do with 150 old servers. The last time we did this, we had 300 old servers. We loaded them on a freight container and they ended up in Bulgaria, where we sold them (we flooded the market with cheap, older servers), and we donated the money to a local professional high school. However, there were unexpected side effects – I used highly-paid professionals for uncharacteristic tasks, I spent over $10,000 to pack and ship the servers, and finally, many of these older machines ended up resold to government institutions, with every imaginable consequence – someone bought old servers for $100 a piece, which was then delivered to government institutions on government contracts, with the intent to be used for critical tasks. And when one of these machines “dies,” thousands of citizens suffer from the lack of administrative services. This is the main reason why this time I decided against contributing to the decline of public Internet services in Bulgaria, and to recycle the servers in the USA instead.
Through the years, I have gradually learned not to compromise, which is why a lot of people consider me eccentric, or outright crazy. Still, I’d like to share with you that living without compromising is wonderful, and I am happy.