Price comparison website Comparethemarket.com is migrating from Microsoft SQL Server to MongoDB as its business grows. It is reducing its cycle time of idea to production from three weeks to just half an hour.
Matthew Collinge, solutions architect at Comparethemarket.com, puts the company's decision to migrate from Microsoft SQL Server to 10Gen’s MongoDB NoSQL database in the context of its strategic preference for agile software development and open source.
In April last year, the company began to migrate. The programme will be completed in around 18 months’ time.
Database technology to support growth
Collinge says the company’s old system was a direct copy from the group's mainframe, ported to SQL Server and the .Net stack. It had been in place for about four years, and dated from the time when the business was smaller. Comparethemarket.com has doubled in size in the past 18 months, in terms of both website traffic volume and headcount.
The site compares 17 million prices per day. It had a single code base that ran all of the comparison products – house insurance, car insurance, pet insurance, and so on – which put “a real strain on the business”, says Collinge.
More on big data analytics in the insurance industry
Moreover, the 100 or so IT staff are divided into product teams, each of which wants to control of their own release schedule and not be affected by changes made by other teams. For this reason, the insurer chose to split the application into units of functionality.
Its scalability options had been limited to buying more and bigger boxes, creating a “capacity bottleneck”. It believes the move to a NoSQL database will help with that.
The project began in April of last year, with the home insurance comparison business. That will now become the platform for all Comparethemarket's business.
Supporting agile development
Collinge explains that since old the system had been around for a long time, a large proportion of the developers’ time had been “taken up by the accidental complexity of the system”.
He says the cycle time it takes from a business person having an idea they want to test on the site to that idea, if a goer, going into production has been reduced from three weeks to 30 minutes.
More on MongoDB NoSQL database
- Betting site YouWin speeds response times with MongoDB database
- How NoSQL, MySQL and MongoDB worked together to solve a big-data problem
- Red Hat thinks big data with MongoDB collaboration
- Solve 'big data' problems with a MySQL cluster, NoSQL
- Cornell Cern project plumps for NoSQL DBMS
- NoSQL technology gains channel support
The fundamental choice was to go for an event-driven architecture to get real-time insight into how the operational system is performing. "That way, we can communicate with an insurance partner as an issue is live, as opposed to waiting for a lot of data to be loaded into a data warehouse for it to be dealt with the following day,” he says.
The new architecture will enable the company to do real-time analytics and real-time personalisation for website visitors, he adds. In the past, for instance, if you wanted to add car radio insurance to your car insurance comparison, it would be done offline.
Collinge says the developer friendliness of MongoDB and its good community support were important factors in the choice. The company has a support arrangement with 10gen. “It is one of the more helpful IT organisations,” he says.
Whether Alexander Orlov, owner of Comparethemeerkat.com, or, more probably, his IT director Sergei, has also chosen a NoSQL database architecture is not known at this time.