To the mesosphere and beyond

Nick Booth finds that the secret to banishing the fail whale could be Mesos

If you use Twitter and Airbnb, you’ll know neither is perfect. I’ve used Airbnb twice now, for trips to Amsterdam and Las Vegas and in both cases, there ‘information problems’. Let’s say that the users didn’t declare all the variables and leave it at that.

Twitter, on the other hand, is often forced to display its default error message – that cutesy picture of a blue whale being hoist out the ocean by eight tiny orange birds. “Too many tweets” says the caption.

I’m surprised South West Trains, Thames Water or BT haven’t copied that idea for explaining system failure. They could have eight cutesy Hi-Viz jacket wearing cartoon characters, straining to pull a train, a pipe or a BT van out of the sea.

The difference between Twitter and many British brands is that the US firm will actually beaver away like crazy to rectify the problems. Whereas British firms think it’s OK to just stick out a schmaltzy apology – on Twitter, ironically enough.

The system that is widely credited with removing the ‘Fail Whale’ from public view, by eradicating all the bottlenecks in Twitter’s production process, is Mesos.

Twitter runs its private data centers on Apache Mesos, as do eBay, Netflix, OpenTable and PayPal.

For developers, Mesos provides simple command-line and application programming interface (API) access to computing clusters. This means applications can be set up, then subsequently have a fitting environment tailored around them, automatically. The system does this, which means that the IT operations plods don’t need to be involved.

This capacity for automated self adjustment could save data centres bezillions of man hours and dollar, according to Peter Levine, the founder of virtualisation pioneers Xensource.

Mesos software was the foundation for some of the most reliable and high-performance datacenters in the world, thanks to this dynamic effect, says Levine. In his new capacity as general partner at venture capitalist Andreessen Horowitz, Levine has helped fund a new plan to use Mesos to create a system that manages a data center like a single machine – a breakthrough that some claim would transform the industry. 


The Mesosphere company, founded by engineers from Twitter and Airbnb, has a simple objective. It wants to speed the adoption of Apache Mesos by providing new enterprise products and commercial support for the open source software. It’s just been given $10.5 million in series A funding from investors including Andreessen Horowitz, Data Collective and Fuel Capital.

By abstracting and automating the most difficult low-level tasks related to a range of critical functions, the software can have a dynamic effect on data centres. The launching and managing of services, virtual machines and containers in expanding datacenter environments, can all be streamlined and automated. This removes many of the production bottlenecks and creates continuity, fault tolerance and better server utilization.


All cloud and data centres become as one.


“Managing your datacenter as if it’s a single computer is the holy grail of cloud computing, and Mesosphere actually delivers on that compelling vision,” says Mesosphere investor Brad Silverberg, who once led Microsoft’s Windows, Office and Internet platform business units.


“This is the direction of the new datacenter and the shift will be as transformational as virtualization was,” says Levine.


Now all they need to do is transform the people who advertise their spare rooms on AirBnb. That’s probably more of a long term problem.

Read more on Independent Software Vendor (ISV) community