How TensorFlow makes Candy Crush virtual players

Simulating a human gamer has enabled Candy Crush developer King to speed up its release cycles.

The evolution of DeepMind’s AlphaGo deep learning algorithm was the inspiration behind mobile games developer King’s work to build a simulation of a games player using Google’s TensorFlow.

AlphaGo beat Go world champion Lee Sedol in 2016. To simulate the ancient game of Go, AlphaGo needed to play the game over and over again, applying a technique called a Monte Carlo search, which uses a deep neural network to “learn” what is the best play move to make.

At the time, artificial intelligence (AI) researcher Demis Hassabis, co-founder of DeepMinds, which Google acquired in 2014, described how open source libraries for numerical computation using data flow graphs, such as TensorFlow, allow researchers to efficiently deploy the computation needed for deep learning algorithms across multiple CPUs or GPUs.

According to GitHub’s Octoverse 2018 report, TensorFlow was by far the most popular open source project in 2018.

King’s team wanted to see if one of the techniques used by AlphaGo could be applied in Candy Crush using TensorFlow. But why should it simulate the game?

Simulation is a key part of the company’s drive to test and release new levels of Candy Crush rapidly. The challenge King wanted to address was how to speed up the work done by games testers in assessing the difficulty of a new Candy Crush level before its release.  

Alex Nodet, an AI engineer at King, said: “We used to have games players play the same level 20 times and write a report. But this took approximately a week and we wanted to be able to release 15 new levels a week.”

The bottom line is this: the more levels King can test, the more it can release. But testing game play has traditionally been done by humans – and it is slow.

To tackle the problem, said Nodet, King began a research project three years ago to develop a machine learning application built using TensorFlow that could test the viability of using a bot to run simulations to canvass how difficult a new level of Candy Crush is to play.

It took about six months to understand the problem that needed to be solved and to create a machine learning model with TensorFlow of a Candy Crush game player using supervised learning. TensorFlow has a steep-learning flow, but there is much more to deep learning that goes beyond competency in the framework tools.

“We did quite a lot of experiments to get it closer to a real player and changed the architecture to build a real player,” said Nodet, adding that the company needed to go through about 20 iterations of the model to get to where it could be used for its intended purpose.

In Nodet’s experience, most of the time spent on a machine learning project is actually not spent tweaking the machine learning model. Instead, the difficult part is getting the data into the right format and then transforming it into something that can be used.

King found that much of the time and effort spent developing new deep learning models involves data cleansing, data labelling and data marshalling.

But training data in only one part of the equation. “Having a machine learning model that is trained and learning is only half the work,” says Philipp Eisen, a machine learning engineer at King. 

Eisen said it is also necessary to consider how machine learning works in production, because machine learning data is something that can have a huge impact on the overall performance of any model. “It must be considered a first-class citizen alongside the network architecture and training algorithms,” he said.

The King team used Cloud Deployment Manager to deploy Google Kubernetes Engine workloads to create hundreds of virtual players, trained with machine learning models. Nodet said: “Our scaling capabilities were not dynamic enough, and Google Kubernetes Engine solved that issue. It is also a great complement to Google Cloud Machine Learning Engine when developing AI applications.”

The resulting data is piped back to King’s data analytics modules with Cloud Pub/Sub, Google’s messaging middleware for the public cloud. This creates a feedback loop enables King to optimise the design of its games to make sure that new levels are not too easy or too difficult, based on a foundation of solid data.

“We have been able to turn our prototype into a production-scale support tool that people want to use,” said Nodet.

According to Nodet, using machine learning, scaled on Google’s G-Cloud, now enables the company to collect feedback on a new Candy Crush level in just five minutes.