James Thew - Fotolia

Spotify to move datacentre workloads to Google’s cloud

Spotify says building and buying datacentres to keep up with demand for its services is no longer viable, as it prepares to shift its infrastructure into the Google cloud

Spotify is to shift its IT infrastructure to Google Cloud Platform and wind down its reliance on private datacentres to keep up with the growing demand for its music streaming services.

The music service is reportedly used by around by 75 million active users in 58 countries. Spotify lets users stream and listen to songs through mobile devices, desktop PCs and gaming consoles.

In a blog post outlining the move, Nicholas Harteau, vice president of engineering and infrastructure at Spotify, said the company had previously bought or leased datacentre space to provide users with local access to the 30 million songs in its catalogue.

However, as the Spotify user base has grown, meeting the growing demand for its services through datacentres has proven difficult, prompting the company to rethink its stance on cloud.

“Operating our own datacentres may be a pain, but the core cloud services were not at a level of quality, performance and cost that would make cloud a significantly better option for Spotify in the long run,” said Harteau.

“Recently that balance has shifted. The storage, compute and network services available from cloud providers are as high quality, high performance and low cost as what the traditional approach provides. This makes the move to the cloud a no-brainer for us.”

Read more about public cloud

The company has decided to shift its infrastructure onto Google Cloud Platform, with Harteau talking up the range of services Spotify will have access to from a data processing and analytics perspective.

For instance, having access to Google’s BigQuery and Cloud Dataproc tools will let Spotify run complex data queries in minutes, rather than hours, to support its product development activities.

Services pinpointed

Spotify stopped short of outlining how long it expects the move off-premise to take. Guillaume Leygues, lead sales engineer for Google Cloud Platform, explained in a blog post the size of the task at hand and the range of cloud services it will adopt along the way.

The company intends to make use of Google Cloud Datastore and Google Cloud Bigtable to fulfil its storage requirements, as well as the cloud service’s Direct Peering, Cloud VPN and Cloud Router networking technologies.

“On the data side of things, the company is adopting an entirely new technology stack,” added Leygues. “This includes moving from Hadoop, MapReduce, Hive and a series of home-grown dashboarding tools, and adopting the latest in data-processing tools, including Google Cloud Pub/Sub, Google Cloud Dataflow, Google BigQuery and Google Cloud Dataproc.”

AWS history

Previously Spotify had been lauded by Amazon Web Services (AWS) as a reference customer, with its use of the Amazon Simple Storage Service (S3) previously the subject of an AWS case study.

Computer Weekly understands that Spotify’s use of AWS for the workloads outlined in the case study is set to continue, while Google Cloud Platform will play host to Spotify’s more “traditional” datacentre infrastructure. 

Computer Weekly contacted Spotify for further comment on this point, but had not received a response at the time of publication.

News of Spotify’s move to the Google cloud comes hot on the heels of Netflix’s announcement earlier this month about its completion of a seven-year push to shift the bulk of its IT infrastructure off-premise and into the AWS public cloud.

Read more on Infrastructure-as-a-Service (IaaS)