In a guest blog, Neo Technology’s CEO Emil Eifrem says graph databases are being looked into by government to manage big data.
While classical business database software, RDBMS (relational database management systems), still has an important role to play in the government context, these systems struggle to tackle new types of data-based problem that the public sector in particular wants to get a handle on.
Why? Relational databases are adept at managing transactional and analytical requirements and are easy to set up, access and extend. But they are challenged by the large amounts of data that agencies now need to manage, specifically in the context of the connections between data that so much of the real world is based on.
Public services – especially the new class of digitally-enhanced or delivered ones governments want to see – depend on being able to spot these connections for legal or improved service delivery reasons. In response, a way of working with data, graph database technology, is emerging as the tool that could help the public sector for this class of applications.
The NoSQL contribution
Graph databases are a big part of a new generation of database technologies for managing large datasets out of the NoSQL family. NoSQL (‘Not SQL’) includes the key-value store, the column family database, the document database and the graph database, while a fifth technology, Big Data data stores like Hadoop, oriented at large-scale batch analytics, has also emerged.
Each of the growing band of post-RDBMS databases has different strengths, but all are aimed at harnessing large volumes of data better than their SQL forebears. That matters, as we are all generating more and more data every day – a data mountain that’s a major challenge for government looking to gain actionable insight into issues.
However, while NoSQL can power all sort of big data work, for tasks that require examining the connections between people, places and events in the real world the tool that can help is graph technology. That’s because graph databases are fantastic at handling both data connectedness even with huge amounts of data. With graph databases civil servants can start to see patterns emerge by connecting multiple legal, welfare and demographic datasets thanks to the connections graph technology highlights/become apparent with – and which are starting to translate into potentially innovative new ways of helping us, the digital citizen.
Let’s consider a real example of how graph database technology is enabling functionality that RDBMS and Big Data/Hadoop technology would struggle to produce. A G8 country needed help in better visualising the relationships and connections powering modern social life. In this case, this was a case management application, helping its civil servants identify individual cases of potential criminal interest, including national security, immigration abuse (visa fraud) or attempts to obtain benefits illegally. Immediate access to such information has been deemed to be the difference between identifying and stopping a criminal and leaving it too late – time no-one can afford to lose as a relational database is still spinning away, crunching data too complex for it to process in real-time.
Meantime this particular government customer is deriving insights, which aren’t just helping it deal with this first problem, but which are becoming the data-driven basis for truly informed, data-based policy creation. As a result, graph technology is starting to enable a new, highly responsive informal learning system for this government – a move with major implications for not just it, but all e-government work.
Judging by these experiences and the kinds of conversations we have with public sector ICT and policy people, we believe graph databases could make a contribution in the drive to deliver cheaper, more integrated, and richer digital public services.
The author is co-founder and CEO of Neo Technology, the company behind Neo4j (http://neo4j.com/).