It has long been acknowledged that there is value in mining data gathered during the course of doing business. The reasoning goes that there are gems in this information that can deliver real value. For example, such data can predict trends and provide partners and clients with tailored information and in general provide insights that are not readily apparent.
Emergence of new paradigms and technologies also has implications from a security perspective. Big data is increasingly seen as an enabler for intelligence-led security, a term that has gained wider visibility during 2012.
Widespread approaches to security-related data analysis have focused on specific assets, namely those able to integrate with whichever formats are supported by the correlation products in use. This necessarily means there are also constraints in what information can be collected, as well as volumes.
It is however recognised that security-pertinent information may be obtained from other areas. Potentially useful information can be obtained by collating all data and appropriately mining it.
In this sense, big data technologies start to become an attractive proposition, given the ability to derive intelligence from internally captured information which can be treated in a reasonable amount of time with a view to identify patterns of interest.
This is particularly interesting in regard to analysing activity on the network over extended periods of time, which can yield information that is not readily apparent otherwise. Think of identifying potential correlations that are not known in advance and factoring that into overall correlation.
While it is becoming common to discuss big data as a potential replacement for security information and event management (SIEM), in reality they should co-exist: SIEM providing the real time alerting and newer technologies using this as a feed to establish wider context, be it for analysing what went wrong as well as defining new rules and metrics.
Armando Leite is a member of London Chapter ISACA Security Advisory Group