As we get into the swing for a new year, it promises to be even more interesting than last year in terms of databases and BI.
On the positive side, the big thing for 2010 was the amount of data being moved into the Cloud. This was helped by increasingly aggressive marketing campaigns from Cloud vendors and the promise of lower costs for business. We also saw an increased interest in using virtualisation for business critical databases and applications although it has a long way to go if it is to match the uptake of virtualisation in the collaboration and email world.
New services which are driven by large amounts of data also appeared. Business Intelligence as a Service (BIaaS) is one example where the big players such as IBM and Salesforce looked to provide more value for customers. But for BIaaS to really work it needs to have the data and the application in the Cloud and that has helped drive data migration.
Another driver has been the launch of Software Testing as a Service (STaaS). While virtualisation has really improved the ability of companies to test their software, there are still limits on how far testing can be scaled. Using a Cloud approach, testing teams can not only keep all their test plans in the Cloud but can add resources to test for extreme circumstances as and when they need them.
We saw the very top end of the database market, with vendors such as Aster Data, Teradata, Greenplum, IBM and Oracle, roll out massively parallel database solutions to deal with the every increasing demand for fast analysis of enormously complex sets of data.
But not everything has been positive. A lot of enterprise operations teams are still concerned about the impact of virtualising their core application servers, especially their databases. Their concerns, which are reasonably founded, are that they do not want spare capacity on their servers being used by other applications. Nor do they want to find their core applications being migrated from one machine to another to meet resource demand. As a core application they want to ensure that their headroom, which allows them to meet sudden increases in demand, is left untouched.
The fall out from the financial crisis has also taken its toll on big database projects with governments, especially in the UK, cancelling a number of the uber databases that were being built. This was partly cost, partly complexity and partly a response to public concern over the amount of data being gathered. A major casualty has been the NHS system which had been beset with a seemingly endless catalogue of problems and mismatched data.
Privacy advocates also weighed in against the rise of the database and BI market. As governments continue to grab all the data they can under the pretext of anti-terrorism laws, it has become increasingly clear that much of this has little do with protecting to public and more to do with empire building and appearing to do something.
There are two main issues here. The first is that the databases have been built to hold discrete sets. By joining them up in order to do very detailed analysis, it puts too much private data into the hands of people who do not need that data. The second issue is that many of the databases are incompatible and the data integration and BI tools are unable to keep up with what they are dealing with.
The constant loss of data from large database, mainly government databases, has also garnered a lot of negative publicity and that is unlikely to go down over the next few years.
Moving to the Cloud has also caused some serious application issues. Part of this is about control and ownership of data and part is down to the problems of latency in applications that are not tolerant enough to cope with applications and data not being co-located.
So will any of that change in 2011? Yes.
One of the big things for 2011 will be a continued consolidation of the database market. Cloud vendors need databases and vendors who want to be enterprise players need an application suite which is generally database driven. Expect to see several vendors acquired as this market shrinks further. We might even see regulators finally force Oracle to divest MySQL in order to put competition back into the market.
It is not just database vendors who will be looking at the Mergers and Acquisitions market. BI vendors will also be looking to see what they are worth and with companies under increasing pressure to make the most of their assets, data mining is a key target area. Cloud vendors also need to keep ahead by offering applications that will make them money so BIaaS which should mean more revenue from virtual machines sales and other services will lead them to invest in this area with acquisition likely to be key for some of the bigger players.
We are already seeing the software development industry revisiting something it thought was done and dusted – latency. Architects are beginning to look at how they can hold data locally to meet privacy and compliance legislation requirements while running the applications in the Cloud. This will mean a rewrite of applications and the creation of more latency friendly solutions. For databases, latency has started to become a serious problem when they are not designed for asynchronous replication.
Another responsibility for developers is how they deal with mobile data. This is not a new problem but the ramp up of mobile applications consuming corporate data has been significant over the last two years. As a result, developers need both encrypted local databases and simple interfaces to ensure that they can sync properly with backend servers.
BI tools are getting easier to use but there is still a paucity of them on the desktop. This is changing and we should see open source start to play a part in the wider tools space. However, vendors such as Oracle with OpenOffice need to make more effort to compete with Microsoft.
A seriously contentious issue is rights management. I dislike it when it is applied to my media to prevent me using it on all my devices but from a data perspective there is a serious need for rights management solutions. As we increase the use of BI then we open up corporate databases to an ever wider pool of people. Security now has to extend from the core database, through the application and down to the devices that are consuming the data. IBM has just shown that it doesn’t quite get this with Cognos 10 and the use of .MHT files but we do need to take this seriously.
International co-operation on data privacy. This goes way beyond databases and BI but there is a need for greater international harmony over data privacy. The current Safe Harbour agreements do not meet many European Government’s compliance legislation requirements and this means that companies are often breaking the law when they move their data offsite. A simplification to make it easier to understand the law would be a major step forward but vendors also need to do more.
Fujitsu has taken the lead over US owned companies as they have put in place mechanisms to protect data from US Patriot Act requests. By comparison, Microsoft has admitted that no matter what it does, customers data will always be subject to Patriot Act requests. IBM, Oracle, HP, Salesforce and other big vendors who are building Cloud database offerings refuse to give any commitment over the security of data stored outside the US. Application vendors need to step up their support for data protection requirements, especially where the data is stored in backups. All backup solutions use a database to track what has been backed up so that it can retrieved. These catalogues have always been the weakest point, after user mishandling of backup media, in the backup process. To meet data protection needs there needs to be a way to mark individual pieces of data as deleted so that even an attempt is made to restore them, they cannot be accessed. The best way is through the catalogue and the database and backup industries need to work more closely together.
There are at least another 10 things on my wish list for 2011 but most of them, like international agreements of privacy and better data protection are a more like tilting at windmills than expecting changes from the industry.
Personally, I’ll settle for just a few mergers and some better tooling.