Adfonic processes 50,000 mobile ads per second with big data architecture

Mobile advertising firm Adfonic serves 50,000 ad impressions per second for clients on smartphones and tablets. SQL, MySQL and NoSQL all play a part

Mobile advertising is a rare beast in the recessionary jungle. Growth from 2011 to 2016 is predicted to be 400%, according to Gartner. 

Mobile advertising platform firm Adfonic, founded at the start of the financial crisis in 2008, is growing in step, says its co-founder and chief technology officer Wes Biggs (pictured).

To serve 50,000 ad impressions per second for clients running advertising on smartphones and tablets, the firm has built a technology architecture that weaves NoSQL, MySQL and SQL databases. It also employs a data science team of four in its London headquarters.

Database advances and increased affordability

Biggs' first job was moving the Los Angeles Times from CompuServe to the internet in 1996. 

“The most exciting thing about being an engineer in this field is that the technology keeps improving and evolving,” he says. And the big change, he adds, has been the dramatic drop in storage costs. 

The usefulness of databases has not changed, but the ability to store high volumes of data has drastically improved in price

Wes Biggs, Adfonic

“The usefulness of databases has not changed, but what has changed is that the ability to store high volumes of data has drastically improved in price. It is now so cost-effective to do that we can think about how to get value out of it. That is really the driver,” says Biggs.

He cites Adfonic's move from fixed-disk database servers to solid state drive (SSD) as an example of this dramatic change in storage cost. “Platters to solid state was a quantum leap forward in terms of performance for Adfonic. Ten years ago, you could not have even bought that on a commercial, widespread basis,” says Biggs.

The company enables advertisers – which include Tesco, Yell, and Peugeot – to bid for display advertising space on mobile sites and applications. It has 3,500 publishers in its network, which serves 100 billion ad impressions every month. 

An increasing aspect of the business is real-time bidding (RTB). “These are instantaneous auctions of ad impressions, with concomitant data in each one,” says Biggs.

More on big data architectures in high-growth industries

Data scientists don’t wear special hats

Adfonic employs around 60 people, in London, New York, Paris, Munich, Madrid and Singapore. Its data science team comprises four PhDs with statistics and artificial intelligence backgrounds. They do not, however, “get to wear special hats”. Instead, among other tasks, they define the algorithms to predict a value for every ad impression based on the probability of it leading to a commercial transaction.

“We don’t have a data science team for the sake of it. They spend a lot of time talking to ad operations about specific campaigns, and they provide the data for our education of the market – our quarterly global AdMetrics report. They are consulting data scientists, as Sherlock Holmes was a consulting detective,” says Biggs.

He describes Adfonic as a “needle in a haystack” operation. “It's a data company, and the nature of the data means that it is big, and there is a lot of it. The term ‘big data’ is, however, very supplier led. It’s just a fancy way of getting at the technology behind business intelligence.” And the point of that is to target the right advert at the right smartphone or tablet user at the right time on behalf of each client.

Biggs explains the company’s data architecture as a combination of SQL, NoSQL and key value stores.

“For the low-latency, high-volume transactions, we make a lot of use of key value stores. For the data processing and real-time reporting back to our clients, we find that SQL is more efficient,” he says. They use a MySQL cluster, availing themselves of Oracle’s MySQL, but also using MySQL replacement MariaDB and Percona’s MySQL. 

The business intelligence (BI) dashboards that Adfonic provides to clients are built in-house, using a Java platform and customised MySQL database, says Biggs.

Download additional resources on business intelligence

NoSQL database deletes expired records

He says the company’s ad server environment means interactions with customers are very short lived. "Should a user seek to engage, we need to be able to record all data corresponding to that impression. But 99% never lead to a click. This means we have to cache a lot of data, but we also want it to expire. We found that traditional caching technology is not geared for expiration.”

After experimenting with the memory object caching system Memcached, Adfonic chose, in late 2012, a NoSQL database from Aerospike, then called Citrus Leaf. This database can remove expired records through its Evictor automated feature. “Our other option,” says Biggs, “was writing an application to delete operations, which would require more effort and more processing time.”

Biggs says his development team, which he feels is about the right size for what the company needs to do, will continue to look at big data technologies as they evolve, such are imperatives of the high-growth market it operates in. 

Analyst firm Gartner predicts that worldwide mobile advertising revenue will reach $11.4bn in 2013, up from $9.6bn in 2012. The ever-growing uptake of smartphones and tablets will conspire with the continued decline of print advertising to stoke this growth. “It’s a good place to be,” says Biggs. “I count my blessings.”

Read more on Business intelligence and analytics